Error Executing the Runbook in System Center Orchestrator

Issues:

Runbook is not getting triggered and also when I try to open orchestrator web console I see the below error

So, I went with the basic troubleshooting steps

  • Checking if the orchestrator service account is not locked out
  • Cross checked, that account has all the access, it is admin on the server and also the same account has been used during setup and configuration

All looks fine except in event viewer I find an error which says access denied for orchestrator account when a runbook is getting triggered.

Went and checked all the possible things including IIS and network settings, and ultimately understood it is happening as Orchestrator web service is broken and as a result web service is not calling the runbooks.

Resolution:

Reinstall Orchestrator web service only and everything will work as expected.

 

Advertisements

Unable to add a second SCOM 2016 MS: Event ID 1008

Recently i came across a strange situation where i was unable to add a second management server to the existing scom management group.

I had taken care of all the pre-requisites like:

  • SCOM actions, SCOM sdk were admins on the second management servers as well as on the DB servers
  • Even the account and computers accounts were admin on both RMS and DBs and DW
  • Firewall ports are opened and traffic is allowed

still the second SCOM MS would not get installed

SCOM secondary MS setup error

It will fail at the stage of Data ware house configuration and roll back everything.

 

Now, the only error in event id under Application log is:

The Open Procedure for service “MOMConnector” in DLL “D:\Program Files\Microsoft System Center 2016\Operations Manager\Server\MOMConnectorPerformance.dll” failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

Event ID 1008 Source: Perflib

i researched online a lot but nothing solved. until i got another error in the scom setup log file.

Below is part of the error, for you to understand the context

15:38:43]: Info: :Finished evaluation of rule ‘NewDBForConfigureDataWarehouseForAllServersRules’

15:38:43]: Info: :Finished evaluation of rule ‘NewDBForConfigureDataWarehouseForAllServersRules’

[15:38:43]: Debug: :Action ConfigureDataWarehouseForAllServers will not be needed.[15:38:43]: Always: :Done validating action list; now running individual actions.[15:38:43]: Always: :Current Action: GetCommonProperties

[15:38:43]: Info: :Info:Getting Common Values for Server Postprocessor

[15:38:43]: Info: :GetCommonProperties completed.

[15:38:43]: Always: :Current Action: StartServices

[15:38:43]: Always: :Starting OM Services.

[15:38:43]: Debug: :StartService: attempting to start service OMSDK

[15:38:43]: Debug: :StartService: Able to start the service OMSDK after 0 minutes.[15:38:43]: Debug: :StartService: attempting to start service healthservice

[15:38:43]: Debug: :StartService: Able to start the service healthservice after 0 minutes.[15:38:43]: Debug: :StartService: attempting to start service cshost

[15:38:43]: Debug: :StartService: Able to start the service cshost after 0 minutes.[15:38:43]: Info: :StartServices completed.[15:38:43]: Always: :Current Action: GetDataReaderWriterAccounts

[15:38:47]: Error: :GetAccountForAProfileFromManagementGroup error: Threw Exception.Type: System.InvalidOperationException, Exception Error Code: 0x80131509, Exception.Message: Sequence contains no matching element

[15:38:47]: Error: :StackTrace:   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)   at Microsoft.EnterpriseManagement.OperationsManager.Setup.ReportingComponent.GetAccountForAProfileFromManagementGroup(ManagementGroup managementGroup, String profileGuid, Guid managementTypeId, String& userName, String& userDomain)[15:38:47]: Error: :GetDataReaderWriterAccounts failed with the following exception: : Threw Exception.Type: System.Reflection.TargetInvocationException, Exception Error Code: 0x80131604, Exception.Message: Exception has been thrown by the target of an invocation.

[15:38:47]: Error: :StackTrace:   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)   at System.Delegate.DynamicInvokeImpl(Object[] args)   at Microsoft.EnterpriseManagement.SetupFramework.ActionEngine.Action.Run(String displayStringNamespace, ProgressData progressData, Func`2 progressDelegate)   at Microsoft.EnterpriseManagement.SetupFramework.ActionEngine.InstallStep.Run(String displayStringNamespace, ProgressData progressData, Func`2 progressDelegate)[15:38:47]: Error: :Inner Exception.Type: System.InvalidOperationException, Exception Error Code: 0x80131604, Exception.Message: Sequence contains no matching element

[15:38:47]: Error: :InnerException.StackTrace:   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)   at Microsoft.EnterpriseManagement.OperationsManager.Setup.ReportingComponent.GetAccountForAProfileFromManagementGroup(ManagementGroup managementGroup, String profileGuid, Guid managementTypeId, String& userName, String& userDomain)   at Microsoft.EnterpriseManagement.OperationsManager.Setup.ReportingComponent.GetDWWriterAccountFromManagementGroup(String managementServerName, String& userName, String& userDomain)   at Microsoft.SystemCenter.Essentials.SetupFramework.InstallItemsDelegates.OMDataWarehouseProcessor.GetDataReaderWriterAccounts()

[15:38:47]: Error: :FATAL ACTION: GetDataReaderWriterAccounts

[15:38:47]: Error: :FATAL ACTION: DWInstallActionsPostProcessor

[15:38:47]: Error: :ProcessInstalls: Running the PostProcessDelegate returned false.[15:38:47]: Always: :SetErrorType: Setting VitalFailure. currentInstallItem: Data Warehouse Configuration

[15:38:47]: Error: :ProcessInstalls: Running the PostProcessDelegate for OMDATAWAREHOUSE failed…. This is a fatal item.  Setting rollback.[15:38:47]: Info: :SetProgressScreen: FinishMinorStep.[15:38:47]: Always: :!***** Installing: POSTINSTALL ***[15:38:47]: Info: :ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs

After Binging a lot i still did not find any solution or any blog where something similar was mentioned.

My usual mantra is to go over and over the error logs in most cases it can direct you to the hidden cause. Well in this case also, it did for me.

Actual Issue:

when i read the setup error log i understood that scom setup while was unable to get the database reader account details and it is something mandatory for it to process and complete the setup.

Now, comes the resolution part:

Go to SCOM console – > Administration pane -> run as profile and check for the accounts associated for below two:

SCOM DW Run as account.png

In my case the associations of run as profile with scom DW account was not targeted properly. I added all the necessary targets as shown in below screenshots.

SCOM Run as Profile association target 2SCOM DW run as profile asscoated target 1

Once it was added, i rebooted the server.

Now, before running the setup again, you will need to delete the management server entry manually from the RMS.

Note:  There will be no entry of SCOM on the server, but if you login to RMS you will see an entry of the MS in grayed out state.

This happens as it was able to successfully register with the root management server during the setup and while it failed to register with data warehouse, scom setup file cannot rollback or unregister itself.

Now rerun the setup and it will be successful.

Voila! issue solved.

Unix / Linux SCOM Commandlets

CMdLet Description
Get-SCXAgent Returns list of managed UNIX / Linux computers
Get-SCXSSHCredential Creates an SSH credential
Install-SCXAgent Install SCOM agent for discovered UNIX / Linux computers.
Invoke-SCXDiscovery Invokes the discovery operation for the specified configuration of UNIX / Linux computers.
Remove-SCXAgent Remove a UNIX or Linux computer from a management group.
Set-SCXResourcePool Change the managing resource pool for the targeted UNIX or Lunix computer.
Uninstall-SCXAgent Uninstall the UNIX / Linux agent.
Update-SCXAgent Updates the UNIX / Linux agent
scxcertconfig -list List the Xplat certificates installed in management group
scxcertconfig -remove Remove the Xplat certificates installed in management group

Example 1:

Input: get-SCXagent

Output: Will return list of all Unix / Linux managed agents

Example 2:

Input: get-SCXagent | where {$_.Name -match “X01C-XPSCOM”} | Remove-SCXAgent

Output: No output will be displayed however, agent that matches with name X01C-XPSCOM will be removed from management group.

Example 3:

Input: scxcertconfig -list

Output: Will display all Xplat certificates installed in management group.

Example 4:

Input : scxcertconfig -remove-all

Output: No output will be displayed however, all Xplat certificates installed in management group will be removed.

MS SCSM VS Atlasian JIRA Service Desk

Jira SDM SCSM
Overview This is built on Atlassians JIRA workflow engine, JIRA Service Desk offers a collaborative, agile platform and knowledge base that’s low cost, easy to set up. Tool OOTB not fully ITIL compliant This is a product from Microsoft and come with System Center suite. Currently this is in its 3rd generation. Tool OOTB is MOF and ITIL compliant. Offers greater flexibility in terms of integration with other system center and MS products
Ease of use Easy to set up and use. However limited OOTB capabilities, needs a lot of customizations. Easy to set up and use, 2016 edition has better UI experience that previous versions. Also third part applications like ItnetX SCSM Portal, and Cireson SCSM Portal increases UI experience
ITIL Process OOTB only Incident and service request Management, rest modules can be built. However requires effort. OOTB has Incident, service catalogue, change, problem, request fulfillment, release management process
Scalability At the very begining you need to decide what are the features that you will use as this related to design of the solution. Scalability is biggest advantage of SCSM, you can easily scale horizontally or vertically based on your needs.
Automation Capabilities Limited, when it comes to outside Atlasion solutions Enhanced automation capabilities with System center and other MS products like SharePoint, Exchange, AD and other third party products. Need to use System center orchestrator and other third party adapters from Kelverion
Licensing Costs < US$ 40,000 (Approx., will vary based on region) Licensing is based on core so it depends upon your infrastructure sizing
Hardware costs Runs on most VMs technology like VM ware, Hyper-V, Oracle VM Runs on most VMs technology like VM ware, Hyper-V, Oracle VM
Resource costs Medium Low, Tool is very ease to use, however development costs are high
Company Size SME SME
Support Costs Low Low

Patching by Orchestrator Part -2

In this blog i will share with you script and other resources that you can use to automate patching.

Restart-Computer -ComputerName REMOTE_COMPUTER_NAME -Force – Wait WinRM

This will force server to restart and return a success code only after the server restarts and WinRM service is up and running. This will ensure that the steps that you have arranged to be executed after a server reboot happens correctly and they do not get failed.

Like i said i in my earlier Patching by Orchestrator Part -1 Blog that i am unable to share complete runbooks or scripts that i have used due to NDA with my employer.

If you know of more script or other innovative ways of automating patching then please post them in comments section.

Note: Credit and risk for all the script and links that i have mentioned here goes to their respective authors.

Hex & RGB Color Scale

While designing forms in Service Manage or make changes to colour scheme in SCOM you need hex colour codes.

I needed it while making changes to colour scheme in service manger self service portal and had to go to multiple portals to generate value for each colour. So i decided to put all the codes in excel sheet and publish it for everyone who needs it.

Download file: Hex & RGB Color Scale