Inconsistent and failing SCOrch runbook performance due to old Application deployment runbooks stuck in Run mode.
- Look for odd Opalis errors (see below) is the SCSM log (C:\ProgramData\Microsoft System Center 2012\Orchestrator\RunbookService.exe\Logs)
- Run Orchestrator Health Checker (https://www.petri.com/use-orchestrator-health-checker-troubleshoot-runbooks) to find out what runbooks are monitoring (Run Mode)
- Validate the runbooks in Run Mode to ensure they are still working.
We found some old Application runbooks set to Run Mode (monitoring). Turning those off significantly improved our system stability and eliminated the Opalis error.
SCOrch is an integral part of our IT Automation. Employee provisioning, privileged account system access (Local Admin), and Change Management are a few standardized processes the company depends on each day. Unfortunately, we spend time re-running or correcting random, unexplainable incomplete runbook processes or failures. Despite our best efforts, we’ve been unable to fix this completely, until today.
Using While the Orchestrator Health Checker (https://www.petri.com/use-orchestrator-health-checker-troubleshoot-runbooks) to optimize our SCOrch environment, we noted a number of failed unrecognized runbooks. Reviewing our Orchestrator log (C:\ProgramData\Microsoft System Center 2012\Orchestrator\RunbookService.exe\Logs), we also found this error repeating almost every minute.
2018-01-21 06:56:25  1 Exception caught in long __stdcall WorkflowContextComAdapter::GetCounterValue(unsigned long,wchar_t *,wchar_t *,struct tagVARIANT *,wchar_t **)
void __cdecl StorageCallExecutor::throwChained(const class Opalis::Exception &)
<MsgCode>SCE: ActionServerStorage call failed</MsgCode>
struct std::pair<class _variant_t,bool> __thiscall ResourceValueExtractor::getValue(const class std::basic_string<wchar_t,struct std::char_traits<wchar_t>,class std::allocator<wchar_t> > &) const
Digging further into some of the unrecognized runbooks, a previously installed Application Approval Workflow showed some of the enabled runbooks (running). Note the large green “play” arrow showing that the runbook is in monitor mode (running).
Figure 1 – Location of Application Approval Workflow Runbooks
Figure 2 – Some of the Runbooks in Monitor mode (running)
Simply turning off these runbooks eliminate the error and stabilized our environment.