SQL 2016 SP1 CU7 Enterprise, running on a fully-patched Windows Server 2016 VM hosted in Azure.
We have been using Service Broker and the External Activator service for years, mostly without incident. Recently, however, a problem has started.
The Service Broker External Activator service will go into a Stopping state from which it can't be re-started. The EATrace.log shows the error:
ERRORThe External Activator service is aborting ...
EXCEPTIONERROR = 90, Internal exceptions have occurred when External Activator is runtime checkpointing.
EXCEPTIONDETAILSInner Exception:
EXCEPTIONDETAILSSystem.IO.IOException: The process cannot access the file 'C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog' because it is being used by another process.
The fix at this point has been to terminate the ssbeas.exe process. The service can then be re-started. We have been unable to determine the source of what other process is locking the EARecovery.rlog file, virus checker being the first place we looked but we couldn't find anything there or elsewhere.
As a temporary solution, we have created an alert to notify us when the service goes into the stopping state and then manually re-start the service as above. This condition is occurring inconsistently at intervals anywhere from hours to days.
The most recent occurrence of the problem had an extra wrinkle. After terminating the the ssbeas.exe process the external activator service would not re-start. In the EATrace.log we saw:
ERRORThe External Activator service is aborting ...
EXCEPTIONERROR = 54, The recovery log record is invalid: previous record of id 180667 does not exist.
INFOThe External Activator service is shutting down.
The solution at this point was to clear out the EARecovery.rlog file. After that, the service was successfully re-started.
Questions:
- Has anybody seen anything similar?
- What is the root cause of the EARecovery.rlog file being reported as locked by another process?
- What is the cuase of invalid record in the recovery log?
We have been using Service Broker and the External Activator service for years, mostly without incident. Recently, however, a problem has started.
The Service Broker External Activator service will go into a Stopping state from which it can't be re-started. The EATrace.log shows the error:
ERRORThe External Activator service is aborting ...
EXCEPTIONERROR = 90, Internal exceptions have occurred when External Activator is runtime checkpointing.
EXCEPTIONDETAILSInner Exception:
EXCEPTIONDETAILSSystem.IO.IOException: The process cannot access the file 'C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog' because it is being used by another process.
The fix at this point has been to terminate the ssbeas.exe process. The service can then be re-started. We have been unable to determine the source of what other process is locking the EARecovery.rlog file, virus checker being the first place we looked but we couldn't find anything there or elsewhere.
As a temporary solution, we have created an alert to notify us when the service goes into the stopping state and then manually re-start the service as above. This condition is occurring inconsistently at intervals anywhere from hours to days.
The most recent occurrence of the problem had an extra wrinkle. After terminating the the ssbeas.exe process the external activator service would not re-start. In the EATrace.log we saw:
ERRORThe External Activator service is aborting ...
EXCEPTIONERROR = 54, The recovery log record is invalid: previous record of id 180667 does not exist.
INFOThe External Activator service is shutting down.
The solution at this point was to clear out the EARecovery.rlog file. After that, the service was successfully re-started.
Questions:
- Has anybody seen anything similar?
- What is the root cause of the EARecovery.rlog file being reported as locked by another process?
- What is the cuase of invalid record in the recovery log?