Once in a while I run into problems that are really fun resolving. A colleague  of mine said that he was early today to fix the XenApp servers that did not reboot successfully. He said that since the storage of the Citrix Provisioning server was moved from flash to SAS the XenApp servers sometimes cannot boot. A shortage in IOPS is supposed to be the cause of this trouble. However all Provisioning servers combined consume over a TeraByte of storage. During the day this fast storage does not is nearly idle. Only during boot of the XenApp machines it is used in an appropriate manner. So that is quite costly.

VMWare created CMDlet’s the right way. You can basically do anything you can in GUI via Powershell using the PowerCLI.  The PowerCLI is an incredible tool in automatising stuff. We have RES Automation Manager. So I put the bits together and created A Powershell script in RES Automation Manager that runs every night. This script uses a few parameters which you can see in the code in the top section of the script. The script heavily relies on PowerCLI 6.0 This version works for any vCenter server.

The script connects to a VIserver (in this case the vCenter server). Next it gathers a list of all servers. Then it filters server based on the condition if it is PoweredOn and it matches on Name. Next it gets the VMware Tools status. If the status is NotRunning it will be applicable for a restart. The servers will be rebooted in a staggered way. So every two minutes a server will reboot. There’ll be a second attempt if the first attempt fails. At last it sends an e-mail with all server that fit the condition the first time, the second time and server that did not rebooted well.

I hope you can use it.

# Adds the base cmdlets
Add-PSSnapin VMware.VimAutomation.Core
$Recipients = $[Recipients]
$VIServer = "$[VIServer]"
$NotCondition = "$[NotCondition]"
$Condition = "$[Condition]"
$Username = [Environment]::UserName
$Hostname = $env:COMPUTERNAME
$Mailserver = "$[MailServer]"
$Level = "0"
Connect-VIServer $VIServer

#List bouwen van VM's die voldoen aan de voorwaarden
$List =@(Get-VM | ? {$_.Name -match $Condition -and $_.PowerState -match "PoweredOn" -and $_.Name -notmatch $NotCondition} | Get-VMGuest | ? {$_.State -match "NotRunning"})

#Check of de PowerCLi wel geinstalleerd is
If (((Get-PSSnapin).name -match "VMware.VimAutomation") -eq $Null){$Body = "De computer $Hostname heeft geen PowerCLI geinstalleerd staan. Installeer PowerCLI en probeer het opnieuw "; $Level = "1"}

#Check of er een verbinding is met de VI server
If ($global:DefaultVIServer -eq $null -and $Level -ne "1"){$Body = "Kon geen verbinding krijgen met de VCenter Server: $VIServer. Verander de waarde in het script als de servernaam veranderd is. Controleer of de user $UserName rechten heeft op de VCenter server. Controleer eventueel of de PowerCLI tools op de server wel compatible zijn met de Vcenter Server versie." ; $Level = "2"}

#Geen servers gevonden
If ($list.count -eq "0" -and $Level -eq "0"){$Body = "Er zijn geen server(s) gevonden welke voldoen aan de voorwaarden."}

#Actie bij wel servers gevonden
If ($list.count -gt "0" -and $Level -eq "0"){
$Body = "Er waren "+ $list.count +" server(s) die voldeden aan de voorwaarden. Het gaat om de server(s):`n`n"
$Body = "Er worden "+ $list.count +"servers herstart. De servers zijn:`n`n"
Foreach ($item in $list){$Body = $Body + $Item.vm + "`r"}

#Herstarten van servers
Foreach ($item in $list){Restart-VM -VM $Item.VM -Confirm:$False ; Start-Sleep -s 120}

#Check of er nu geen servers meer zijn die voldoen aan de voorwaarden
$List2 =@(Get-VM | ? {$_.Name -match $Condition -and $_.PowerState -match "PoweredOn" -and $_.Name -notmatch $NotCondition} | Get-VMGuest | ? {$_.State -match "NotRunning"})

#Actie bij wel servers gevonden die nog niet goed herstart zijn
If ($List2.Count -ne "0"){
$Body = $Body + "`n`n Er waren server die nog steeds niet goed herstart waren. De naam van deze server(s) waren:`n`n"
Foreach ($item2 in $list2){$Body = $Body + $Item2.vm + "`r"}
$Body = $Body + "`n`n Er worden geprobeerd de server opnieuw te herstarten."
Foreach ($item2 in $list2){Restart-VM -VM $Item2.VM -Confirm:$False ; Start-Sleep -s 120}
#Laatste check
$List3 =@(Get-VM | ? {$_.Name -match $Condition -and $_.PowerState -match "PoweredOn" -and $_.Name -notmatch $NotCondition} | Get-VMGuest | ? {$_.State -match "NotRunning"})
If ($List3.Count -ne "0"){
$Body = "Er zijn fouten gevonden bij het herstarten van servers. Zie onderaan de e-mail!!" + $Body
$Body = $Body + "`n`n De server die niet herstart zijn na twee pogingen zijn:`n`n"
Foreach ($item3 in $list3){$Body = $Body + $Item3.vm + "`r"}
}}}

#Mail versturen
Send-MailMessage -From "Rebootscript@domain.tld" -Subject "VMs rebooten" -To $Recipients -Body $Body -SmtpServer $MailServer

#Output naar Console in het geval dat er geen mail verstuurd is.
Write-host `n`n Mail Body`n
$Body