Set persistent scratch location with PowerCLI

In common server configurations for VMware ESXi Hosts, the boot device is usually an SD-Card or a USB stick. These devices are cheap, require very low-maintenance and have a long life as long as not much is written to them. One disadvantage of Flash or SD storage devices is, that ESXi scratch locations are not created on them. The reason for this is the potentially limited read/write cycles available. In order to still meet the VMware best practice, this blog post will show how to set a persistent scratch location with PowerCLI.

If the ESXi Server is not able to automatically select a persistent scratch location during startup, a ramdisk at /tmp/scratch/ is created.

Possible persistent scratch locations are:

• The location configured in the /etc/vmware/locker.conf configuration file, set by the ScratchConfig.ConfiguredScratchLocation configuration option, as in this blog post.
• A Fat16 filesystem of at least 4 GB on the Local Boot device.
• A Fat16 filesystem of at least 4 GB on a Local device.
• A VMFS datastore on a local device, in a .locker/ directory.

It is not a fundamental problem if a ramdisk for scratch is used, but VMware highly recommends that ESXi has a persistent scratch location available for storing temporary data including logs, traces, diagnostic information, and system swap.

Major disadvantages of a ramdisk for scratch:

• Additional host memory is consumed
• No log persistence after reboot or crash (e.g. PSOD)
• Limited system swap capability in case of memory contention

Get current scratch configuration

PowerCLI

# Configured
$VMHost | Get-AdvancedSetting -Name "ScratchConfig.ConfiguredScratchLocation" | ft -Autosize

# Current
$VMHost | Get-AdvancedSetting -Name "ScratchConfig.CurrentScratchLocation" | ft -Autosize

Even when you have configured the persistent scratch location by datastore name, you will get the volume UUID as a result.

This small PowerCLI function returns a table to identify volume UUIDs:

function Get-DatastoreIDs {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory=$True, ValueFromPipeline=$True, HelpMessage="ESXi Host")]
        [ValidateNotNullorEmpty()]
            [VMware.VimAutomation.ViCore.Impl.V1.Inventory.InventoryItemImpl] $VmHosts

    )

    begin{
        $MyView = @() 
    }
    
    process {
            
            foreach ($VmHost in $VmHosts) {
                
                $Ds_View = Get-View (Get-View $Vmhost.ID).ConfigManager.StorageSystem

                foreach ($Volume in $Ds_View.FileSystemVolumeInfo.MountInfo) {

                    if ($Volume.Volume.Type -eq "VMFS") {
                        $Report = [PSCustomObject] @{
                            HostName = $VmHost.Name
                            VolName = $Volume.Volume.Name
                            VolUUID = $Volume.Volume.Uuid
                            DiskName = $Volume.Volume.Extent.DiskName
                            }
        
                        $MyView += $Report
                    }

                }
            }
    }

    end{
        $MyView   
    }
    
}

vim-cmd

# Configured
vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation

# Current
vim-cmd hostsvc/advopt/view ScratchConfig.CurrentScratchLocation

vSphere Client

Set persistent scratch location with PowerCLI

I was faced with the challenge to create a persistent scratch location for a huge number of productive VMware ESXi Hosts. I chose the option of a shared VMFS volume for this environment. So the required steps per host were:

1. Enter Maintenance Mode (evacuate VMs)
2. Create a host-individual .locker folder on the datastore
3. Set persistent scratch location
4. Reboot the host and wait until host is up again
5. Exit Maintenance Mode

To limit the impact on the environment I wanted to process one host after the other.

PowerCLI Script

# Parameters
$ClusterName = "cluster01"
$DatastoreName = "scratch00"

# Get Cluster
$Cluster = Get-Cluster -Name $ClusterName

# Disable VM DRS Roles
$Cluster | Get-DrsRule | Set-DrsRule -Enabled:$false

# Disable HA Admissioin Control
$Trash =  $Cluster | Set-Cluster -HAAdmissionControlEnabled:$false -Confirm:$False

foreach ($VMHost in $($Cluster | Get-VMHost )) {
    $Shortname = ($VMHost | Get-VMHostNetwork).HostName
    $Datastore =  $VMhost | Get-Datastore -Name $DatastoreName

    if ($Datastore){
        $DatastoreDriveName = "HostStore_" + $Shortname
        $Datastore | New-DatastoreDrive -Name $DatastoreDriveName | Out-Null
        $Trash = New-Item -Path $($DatastoreDriveName + ":\.locker-$Shortname") -ItemType Directory -Confirm:$false -Force
        Remove-PSDrive -Name $DatastoreDriveName
        "Folder '/vmfs/volumes/$($Datastore.Name)/.locker-$Shortname' created"

        $Trash = $VMHost | Get-AdvancedSetting -Name "ScratchConfig.ConfiguredScratchLocation" | Set-AdvancedSetting -Value "/vmfs/volumes/$($Datastore.Name)/.locker-$Shortname" -Confirm:$false

        $Trash = $VMHost | Set-VMHost -State Maintenance -Evacuate -Confirm:$false
        "$($VMHost.Name) entered Maintenance Mode"

        $Trash = $VMHost | Restart-VMHost -Confirm:$false

        "$($VMHost.Name) is Restarting"
        do {
            Start-Sleep  5
            $ServerState = (Get-VMHost $VMHost.Name).ConnectionState
        }
        while ($ServerState -ne "NotResponding")
        "$($VMHost.Name) is Down"

        # Wait for server to reboot
        do {
            Start-Sleep  5
            $ServerState = (Get-VMHost $VMHost.Name).ConnectionState
        }
        while ($ServerState -ne "Maintenance")
         "$($VMHost.Name) is Up"

        $Trash = $VMHost | Set-VMHost -State Connected -Confirm:$false
        "$($VMHost.Name) exited  Maintenance Mode"
    }
    else{ throw "No Datastore Found"}

    
}

$Cluster | Get-DrsRule | Set-DrsRule -Enabled:$true -ErrorAction SilentlyContinue
$Trash =  $Cluster | Set-Cluster -HAAdmissionControlEnabled:$true -Confirm:$False

In addition to the configuration of the persistent scratch location, the script also prepares the Cluster for efficient host maintenance (DRS Rules, HA Admission Control).

After the VMware ESXi Host hat successfully restarted, the scratch location folder structure should be automatically created:

Metro Storage Cluster and shared scratch volume

This script is designed to set a single datastore as a persistent scratch location with PowerCLI, which means one datastore per vSphere cluster or the whole vSphere datacenter. In case you run a VMware vSphere Metro Storage Cluster setup you probably want to choose a datastore per site.

This snippet shows an example of choosing the datastore by vSphere Tag:

$DatastoreName_DC1 = "Scratch_DC1"

$TagCategory = Get-TagCategory -Name "Location"

foreach ($VMHost in $($Cluster | Get-VMHost )) {
    $Shortname = ($VMHost | Get-VMHostNetwork).HostName
    $HostLocation =  $VMhost | Get-TagAssignment -Category $TagCategory
    if ($HostLocation.Tag -match "DC1") {
        "$($VMHost.Name) is in DC1"
        $Datastore =  $VMhost | Get-Datastore -Name $DatastoreName_DC1
    }
    
    if ($Datastore){
    ...
    }
}

References:

• Creating a persistent scratch location for ESXi 4.x/5.x/6.x (1033696)
• Automating configuration of a scratch location for ESXi with PowerCLI
• VSAN considerations when booting from USB/SD
• VMware vSphere 6.5 Host Resources Deep Dive