I am seeing this error every few minutes in sql error log
[i]04/20/2016 17:06:01,Logon,Unknown,Service Broker login attempt by user 'domain\serviceuser.' failed with error: 'A previously existing connection with the same peer was detected during connection handshake. This connection lost the arbitration and it will
be closed. All traffic will be redirected to the previously existing connection. This is an informational message only. No user action is required. State 80.'. [CLIENT: 192.168.50.74][/i]
The environment is a four node WSFC with three SQL Server 2012 SP3 enterprise instances. Under normal circumstances the SQL server roles run on distinct cluster nodes. The physical cluster nodes have IP addresses 192.168.50.71 - 74 and the WSFC
network resources have IP addresses 192.68.50.81 - 83. Each instance is configured to use a static port 50101 - 50103
On each instance service broker is sending messages to another instance. I see the above error in the SQL error log for all of the instances. Each service broker endpoint is configured to listen on all IPs and has port 4022 configured. The
routes addresses are configured as TCP://SQLClusterResourceName:4022.
I made some profiler traces and saw a couple of other errors like "This message could not be delivered because it is duplicate" and
"An error occurred while receiving data: '64(The specified network name is no longer available.)" occurring occasionally.
I'm new to service broker, the initial configuration was done by somebody else, but from doing a lot of web searches it seems that the ACK may not being received by the initiator which cause the message to be resent which is why it's a duplicate, as mentioned
here
social.msdn.microsoft.com/Forums/sqlserver/en-US/91747a32-f221-4d98-990f-af1606ca5c3c/this-message-could-not-be-delivered-because-it-is-duplicate?forum=sqlservicebroker
I found this old post which sounds very similar but unfortunately there's no resolution there.
social.msdn.microsoft.com/Forums/sqlserver/en-US/7cc67a9e-c792-4211-9f17-7dd3c6caf47c/service-broker-error-connection-handshake-failed?forum=sqlservicebroker
I have tried using the FQDN of the SQL cluster resource in the SB route and forcing the listener to the IP address of the cluster resource instead of all. I also suspected a network issue but have found no other symptoms to support that. I have
tried disabling some of the more exotic NIC features, TCP offload, RSS etc. Nothing I tried has made the slightest difference so far.
One thing that surprises me is that the client specified in the error message sometimes has the IP address of the physical cluster node and sometimes the SQL cluster resource IP address, the instances are definitely only listening on the cluster resource IP
address, but the service broker can send using either. Is that correct? Is there a way to force all SQL communication to only send on the virtual server IP?
I tried to uses ssbdiagnose here is the output:
C:\Windows\system32>ssbdiagnose -E RUNTIME -NEW connect to -S server1\instance1 -d dbname1 connect to -S server2\instance2 -d dbname2
Microsoft SQL Server 11.0.2100.60
Service Broker Diagnostic Utility
P 29830 Could not find the connection to the SQ
L Server that corresponds to the routing address tcp://192.168.50.72:4022. Ensure
the tool is connected to this server to allow investigation of runtime events
P 29830 Could not find the connection to the SQ
L Server that corresponds to the routing address tcp://192.168.50.73:4022. Ensure
the tool is connected to this server to allow investigation of runtime events
P 29830 Could not find the connection to the SQ
L Server that corresponds to the routing address tcp://192.168.50.74:4022. Ensure
the tool is connected to this server to allow investigation of runtime events
P 29830 Could not find the connection to the SQ
L Server that corresponds to the routing address tcp://192.168.50.71:4022. Ensure
the tool is connected to this server to allow investigation of runtime events
An internal exception occurred: Access is denied. (Exception from HRESULT: 0x800
70005 (E_ACCESSDENIED))
^C
Can anyone help? Where should I go from here?