Error Executing the Runbook in System Center Orchestrator

Issues:

Runbook is not getting triggered and also when I try to open orchestrator web console I see the below error

So, I went with the basic troubleshooting steps

  • Checking if the orchestrator service account is not locked out
  • Cross checked, that account has all the access, it is admin on the server and also the same account has been used during setup and configuration

All looks fine except in event viewer I find an error which says access denied for orchestrator account when a runbook is getting triggered.

Went and checked all the possible things including IIS and network settings, and ultimately understood it is happening as Orchestrator web service is broken and as a result web service is not calling the runbooks.

Resolution:

Reinstall Orchestrator web service only and everything will work as expected.

 

Advertisements

System Center and Devops

In this blog, I will share with you on how devops approach is followed and maintained while using system center suite of products.

Before going into details let’s talk about devops first.

The definition – DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between development and IT operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between these two business units.

Ref: http://www.webopedia.com/TERM/D/devops_development_operations.html

Now in today’s fast changing enterprise world all business leaders ultimate goal is to be more collaborative and inter connected across various business functions and to do that you need an IT team and technology that enables you.

In market, there are various tools and suite of products that come to help enterprises. However, Microsoft suite of products system center suite is a winner in many areas.

In Devops approach from SME to large enterprise we follow the below approach

System Center DevOps Model

 

Monitor:

Starting with monitoring your entire Infrastructure SCOM (system center operations manager) is a great tool. Why you ask, as OOTB it has all management packs to monitor your entire Microsoft technological solutions and you have plenty of third party solution and adapters that makes it easier to integrate with other technologies or solutions.

Service:

Now when it comes to IT service management, you can rely on SCSM (system center service manager)

It is a great tool to manage all your incidents, service requests, change, problems. Off course you can use it for release and business relationship management but those features are not that great. If you combine it with other solutions then it is wonderful ITSM product.

Manage:

Now for managing your infrastructure you have SCCM (system center configuration manager) OOTB tool can manage all windows software including OS for desktop, laptops, servers and with other third-party solution you can extend its functionality for managing and patching other third-party software’s.

Automation:

Now for a successful devops you need to automate and combine all these functions. This is where SCORC (System Center Orchestrator) along with PowerShell comes in handy. You can automate almost any anything across your infrastructure.

Example Scenarios:

SCOM detects that one of your critical web services is down -> It then automatically create an incident and assigns it to L1 Wintel team. -> Wintel engineer validates the alert runs a runbook in SCORC from the SCSM console (which restarts IIS service) -> Now as the service is started -> SCOM alert is auto resolved and closed -> Incident in SCSM console is resolved with all the actions that got executed in background captured in incident logs -> Wintel engineer notices that from the past incidents and also by his experience that high RAM usage his root cause for this issue-> He goes to SCSM console raises a change request for increasing RAM on the server-> Goes to SCCM console and checks if the server is compliant with all latest security patches and critical updates -> once the change is approved in SCSM-> he uses SCORC runbooks which is integration with SCVMM to increase the RAM on the server -> weeks later from SCOM performance he pulls up a report and verifies that that IIS service going down has never happened after memory was increased on the server.

Above was just a high level example on how all the system center products work hand in hand. This makes it super easy to manage enterprise level IT infrastructure.

Patching by Orchestrator Part -1

Today i will explain you how you can achieve patching by orchestrator including complex patching procedures.

Below is list of software that you will need:

  1. Excel – To put the steps that needs to be carried out in a sequential order
  2. MS SQL DBs
  3. PS Scripts
  4. MS Orchestrator
  5. MS SCCM
  6. MS SCSM (incase you want to make patching a self service offering)

Now a few things to keep things in order:

  • In a Excel sheet arrange all steps that needs to be automated, below is just an example of column headers:

SequenceNo ActionType ComputerName Parameter1 Parameter2 Parameter3 Parameter4 Parameter5 Parameter6 Outcome Expected Patching by Orchestrator Excel sheet Template Patching by Orchestrator Template

  1. Now create DBs called ‘SteptoExecute’ in a SQL instance where you will upload the steps created in the previous steps, my suggestion is to create it in Orchestrator instance itself
  2. Another DBs called ‘ExecutedSteps’
  3. Now you have to create 1 main runbook which in turn will call multiple other runbooks that you need to create in next step.
  4. Here runbooks should be arranged to first read steps from ‘SteptoExecute’ and then invoke runbooks based on sequence#
  5. Other multiple runbooks that you need to create will mainly depend upon complexity of steps, here i will just give a high level of runbooks that will typical be required
    • SCCM Patch push code runbook
    • SCCM patching status code check runbook
    • Service Start runbook – Runbook will take service name
    • Service Stop runbook
    • Service restart runbook
    • Computer restart
    • Computer restart with timeout
    • Computer restart with check when it’s back online
    • Run Program- Can use to run a batch file
    • Web Status code checker – can be used to check response status code of web application – Click to download runbook  web application response status code checker
    • Email activity runbook – to send status of steps executed over an email
  6. Now use orchestrator runbook to invoke steps from the DBs in the sequential order arranged, use SQL DB read activity or a PS script
  7. Each step has an associated action type with it as explained earlier, which in turn will call the runbooks and runbooks will execute the steps such as stop/start/restart services, run batch files, rename files, start a web service, stop a web site and etc
  8. Make sure to make runbooks names same as action type, it will easier to invoke runbooks in the previous step
  9. Once all the steps are executed a email activity combines all the steps executed along with the results that are saved in ‘ExecutedSteps’ Db

Download link to Visio Diagram

In the next part i will share with you scripts that you will need and reference to other MVP and system center blogs which will be helpful to you.

However, due to NDA with my ex employer i will not be able to share complete runbooks or scripts that i have used. 

Please note, do test your runbooks in a dev environment first. Sometimes it can take a few trial and error in getting the steps in a sequence order.

Patching by Orchestrator Part -2

Stop runbook instance by orchestrator

If you are an automation geek, you will come across multiple scenarios where you would like to stop a runbook while it is getting executed. unfortunately OOTB MS orchestrator does not have any activity that supports this model.

There is PS command lets that you can use but it is quite complex and most important it is very difficult to  have it triggered automatically (runbook runtime).

However, there is awesome ready made integration pack available from Kelverion Kelverion integration pack for runbook management you can use this IP to stop, start, get runbook status and get runbook ID. Obviously this comes at a cost, now if you are an geek like me who like to do things rather than buy them then click here: Stop runbook instance

How it works:

Just enter  Runbook ID or unique ID is taken as an input parameter, (you need to fetch this info the orchestraor database or orchestrator web URL).

Once unique ID is entered,  stop runbook will automatically fetch it instance id (this will be unique every time a runbook runs) from orchestrator DB and stop only that instance of the runbook.

I hope this will be of  help to the community.

Please post your feedback in comments section below.