Windows Server 2016 Standard fully patched
We have been using Service Broker External Activator for a few years now with few problems but are having a recurring problem in the past week where the service will abort with the following error in EATrace.log:
18/07/2018 09:59:45 EXCEPTION ERROR = 90, Internal exceptions have occurred when External Activator is runtime checkpointing. 18/07/2018 09:59:45 EXCEPTIONDETAILS Inner Exception: 18/07/2018 09:59:45 EXCEPTIONDETAILS System.IO.IOException: Cannot create a file when that file already exists. 18/07/2018 09:59:45 EXCEPTIONDETAILS 18/07/2018 09:59:45 EXCEPTIONDETAILS at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) 18/07/2018 09:59:45 EXCEPTIONDETAILS at System.IO.File.Move(String sourceFileName, String destFileName) 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.LogManager.SaveRecoveryContext(LogRecoveryContext recoveryContext) 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.LogManager.Checkpoint(LogRecoveryContext recoveryContext) 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.LogManager.Log(LogRecord recoveryLogRec) 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.ApplicationMonitor.OnProcessExited(ProcessMonitor processMonitor) 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.ProcessMonitor.NotifySubscriber() 18/07/2018 09:59:45 EXCEPTIONDETAILS at ExternalActivator.ProcessMonitor.OnProcessExited(Object a, EventArgs b)
When checking the log folder after this has occurred I can see that the EARecovery.rlog file has been deleted and in it's place is an EARecovery_temp.rlog file. I assume that the File.Move call is to rename that to the original name. Nothing new has been installed on this server save for the usual Windows updates.
This started happening on the 16th July and has happened at least once a day since at various times. It does seem to be checkpointing a lot more than I expected though as can be shown from the following snippet from the log:
23/07/2018 11:40:09 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[PDFServiceTargetQueue]. 23/07/2018 11:40:09 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.PDFService\Company.Rubicon.ServiceBroker.PDFService.exe was created: id = 55036. 23/07/2018 11:40:14 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[EmailTargetQueue]. 23/07/2018 11:40:14 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.Emailer\Company.Rubicon.ServiceBroker.Emailer.exe was created: id = 5544. 23/07/2018 11:40:17 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:40:17 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:40:17 VERBOSE Application process id = 55036 has exited: exit code = 0, exit time = 7/23/2018 11:40:17 AM. 23/07/2018 11:40:18 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:40:18 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:40:18 VERBOSE Application process id = 5544 has exited: exit code = 0, exit time = 7/23/2018 11:40:18 AM. 23/07/2018 11:40:29 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[EmailTargetQueue]. 23/07/2018 11:40:29 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.Emailer\Company.Rubicon.ServiceBroker.Emailer.exe was created: id = 56352. 23/07/2018 11:40:33 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:40:33 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:40:33 VERBOSE Application process id = 56352 has exited: exit code = 0, exit time = 7/23/2018 11:40:33 AM. 23/07/2018 11:41:27 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[EmailTargetQueue]. 23/07/2018 11:41:27 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.Emailer\Company.Rubicon.ServiceBroker.Emailer.exe was created: id = 56872. 23/07/2018 11:41:31 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:41:31 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:41:31 VERBOSE Application process id = 56872 has exited: exit code = 0, exit time = 7/23/2018 11:41:31 AM. 23/07/2018 11:41:43 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[PDFServiceTargetQueue]. 23/07/2018 11:41:43 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.PDFService\Company.Rubicon.ServiceBroker.PDFService.exe was created: id = 55892. 23/07/2018 11:41:46 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[EmailTargetQueue]. 23/07/2018 11:41:46 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.Emailer\Company.Rubicon.ServiceBroker.Emailer.exe was created: id = 54584. 23/07/2018 11:41:47 VERBOSE Received event notification for [UKSQL4].[Rubicon].[Brkr].[PDFServiceTargetQueue]. 23/07/2018 11:41:47 VERBOSE Application process C:\Program Files\Company\Company.Rubicon.ServiceBroker.PDFService\Company.Rubicon.ServiceBroker.PDFService.exe was created: id = 27984. 23/07/2018 11:41:53 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:41:53 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:41:53 VERBOSE Application process id = 54584 has exited: exit code = 0, exit time = 7/23/2018 11:41:53 AM. 23/07/2018 11:41:54 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:41:54 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:41:54 VERBOSE Application process id = 55892 has exited: exit code = 0, exit time = 7/23/2018 11:41:54 AM. 23/07/2018 11:41:55 VERBOSE Checkpointing recovery log C:\Program Files\Service Broker\External Activator\log\EARecovery.rlog ... 23/07/2018 11:41:55 VERBOSE Checkpointing recovery log completed. 23/07/2018 11:41:55 VERBOSE Application process id = 27984 has exited: exit code = 0, exit time = 7/23/2018 11:41:55 AM.
Does this look to be a normal amount of Checkpointing? Does anyone have any suggestions on how to track down the cause of the error?