Overprovisioning is a huge topic if you need to run a efficient environment. Especial Storage Overprovisioning has a lot of positive but also a few negative aspects, if you not have a good monitoring concept you will see more negative aspects. This article will guide you through my implementation of a Datastore Overprovisioning monitoring with VMware vRealize Operations Manager.
The problem of Datastore Overprovisioning
Storage Overprovisioning is necessary to leverage the expensive Storage resources in a efficient way. If you do more aggressive Datastore Overprovisioning you are also more efficient but on the other side you have a huge risk. The main technique that alows Datastore Overprovisioning is Thin Provisioning which can be done on VMware VMDK layer and on the Storage layer. If you do Thin Provisioning on both layers you are very efficient but without a proper monitoring you also have a huge risk.
Let’s explain the risk of Datastore Overprovisioning with a simple example:
You have two VMs with a 10 TB VMDK each stored on a 10 TB VMware Datastore. If you deploy these VMDK’s as thin provisioned this situation is a valid scenario. During the first few weeks both VM’s uses for example about 40% of their Disk space but then a database restore happens and one VM uses now 80% of the Disk Space. ⇒ You will run out of space on the Datastore and two VM’s are in trouble
This point brings me to the second metric which describes together with the Overprovisioning-Rate the scale of risk with Datastore Overprovisioning, the number of VM’s on a Datastore. The number of VM’s has impact to the risk in two ways, the first impact is the probability that a huge write will happen and the second one is how many VMs are impacted by a possible fault.
Let me extend my example with another layer of thin provisioning:
You have two of these 10 TB VMware Datastore with a 200% Overprovisioning-Rate on a storage array with 10 TB total capacity. This situation is also a valid scenario if your LUN’s or Shares are thin provisioned, but now you have a huge risk on zwo layers. You can run out of Space on the VMware Datastore and on your whole Storage Array, which has impact to all VM’s.
Datastore Overprovisioning Rule:
- Set a Limit for the Overprovisioning-Rate
- Set a Limit for the number of VMs on a overprovisioned datastore
- Monitor these limits on all layers
Monitoring Datastore Overprovisioning with vRealize Operations Manager
Super Metric
At first we need to create new Super Metric for the Overprovisioning-Rate. All other objects (Views, Alerts, etc.) are related to this Metric.
If that is your first Super Metric you should also check these great Blog Post:
1
|
(${this, metric=diskspace|provisioned}/${this, metric=diskspace|capacity})*100
|
Afterward we need to define the Object Type which the Super Metric applies to.
You can also import this JSON file as new Super Metric:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
{
"b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa":{
"resourceKinds":[
{
"resourceKindKey":"Datastore",
"adapterKindKey":"VMWARE"
}
],
"name":"Datastore OverProv %",
"formula":"(${this, metric=diskspace|provisioned}/${this, metric=diskspace|capacity})*100",
"description":""
}
}
|
More Details: VMware Docs - Super Metrics Tab
Collect Super Metric
Only when the new Super Metric is enabled for collection in your Policy we are able to use and see the Metric on the related objects (KPI and DT is optional).
More Details: VMware Docs - Collect Metrics and Properties Details
Create View
With the prior configuration steps we can now proceed to create a new View to visualize the Datastore Overprovisioning in a nice Table.
View Data:
-
Disk Space|Capacity (TB)
-
Disk Space|Total used (TB)
-
Disk Space|Provisioned Space (TB)
-
Super Metric|Datastore OverProv %
-
Summary|Total Number of VMs
View Filter:
In my Case all local Datastores should be removed from the View.
You can also import this XML file to create the new View:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Content>
<Views>
<ViewDef id="849cafc9-bf05-4860-9afc-114e96575682">
<Title>Datastore Provisioning</Title>
<Description/>
<SubjectType adapterKind="VMWARE" filter="[[{"filterType":"properties","metricKey":"summary|isLocal","condition":"NOT_EQUALS","metricValue":{"value":"true","isStringMetric":true}},{"filterType":"properties","metricKey":"summary|type","condition":"NOT_EQUALS","metricValue":{"value":"VMFS","isStringMetric":true}}]]" resourceKind="Datastore" type="descendant"/>
<SubjectType adapterKind="VMWARE" filter="[[{"filterType":"properties","metricKey":"summary|isLocal","condition":"NOT_EQUALS","metricValue":{"value":"true","isStringMetric":true}},{"filterType":"properties","metricKey":"summary|type","condition":"NOT_EQUALS","metricValue":{"value":"VMFS","isStringMetric":true}}]]" resourceKind="Datastore" type="self"/>
<Usage>dashboard</Usage>
<Usage>report</Usage>
<Usage>details</Usage>
<Usage>content</Usage>
<Controls>
<Control id="time-interval-selector_id_195" type="time-interval-selector" visible="false">
<Property name="advancedTimeMode" value="false"/>
<Property name="unit" value="DAYS"/>
<Property name="count" value="7"/>
</Control>
<Control id="attributes-selector_id_196" type="attributes-selector" visible="false">
<Property name="attributeInfos">
<List>
<Item>
<Value>
<Property name="objectType" value="RESOURCE"/>
<Property name="attributeKey" value="diskspace|capacity"/>
<Property name="preferredUnitId" value="tb"/>
<Property name="isStringAttribute" value="false"/>
<Property name="adapterKind" value="VMWARE"/>
<Property name="resourceKind" value="Datastore"/>
<Property name="rollUpType" value="NONE"/>
<Property name="rollUpCount" value="0"/>
<Property name="transformations">
<List>
<Item value="LAST"/>
</List>
</Property>
<Property name="isProperty" value="false"/>
<Property name="displayName" value="Total Capacity"/>
</Value>
</Item>
<Item>
<Value>
<Property name="objectType" value="RESOURCE"/>
<Property name="attributeKey" value="diskspace|disktotal"/>
<Property name="preferredUnitId" value="tb"/>
<Property name="isStringAttribute" value="false"/>
<Property name="adapterKind" value="VMWARE"/>
<Property name="resourceKind" value="Datastore"/>
<Property name="rollUpType" value="NONE"/>
<Property name="rollUpCount" value="0"/>
<Property name="transformations">
<List>
<Item value="LAST"/>
</List>
</Property>
<Property name="isProperty" value="false"/>
<Property name="displayName" value="Used Space"/>
</Value>
</Item>
<Item>
<Value>
<Property name="objectType" value="RESOURCE"/>
<Property name="attributeKey" value="diskspace|provisioned"/>
<Property name="preferredUnitId" value="tb"/>
<Property name="isStringAttribute" value="false"/>
<Property name="adapterKind" value="VMWARE"/>
<Property name="resourceKind" value="Datastore"/>
<Property name="rollUpType" value="NONE"/>
<Property name="rollUpCount" value="0"/>
<Property name="transformations">
<List>
<Item value="LAST"/>
</List>
</Property>
<Property name="isProperty" value="false"/>
<Property name="displayName" value="Provisioned Space"/>
</Value>
</Item>
<Item>
<Value>
<Property name="objectType" value="RESOURCE"/>
<Property name="attributeKey" value="Super Metric|sm_b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa"/>
<Property name="isStringAttribute" value="false"/>
<Property name="adapterKind" value="VMWARE"/>
<Property name="resourceKind" value="Datastore"/>
<Property name="rollUpType" value="NONE"/>
<Property name="rollUpCount" value="0"/>
<Property name="transformations">
<List>
<Item value="LAST"/>
</List>
</Property>
<Property name="sortCriteria" value="false"/>
<Property name="isProperty" value="false"/>
<Property name="displayName" value="Provisioned Space %"/>
</Value>
</Item>
<Item>
<Value>
<Property name="objectType" value="RESOURCE"/>
<Property name="attributeKey" value="summary|total_number_vms"/>
<Property name="isStringAttribute" value="false"/>
<Property name="adapterKind" value="VMWARE"/>
<Property name="resourceKind" value="Datastore"/>
<Property name="rollUpType" value="NONE"/>
<Property name="rollUpCount" value="0"/>
<Property name="transformations">
<List>
<Item value="LAST"/>
</List>
</Property>
<Property name="isProperty" value="false"/>
<Property name="displayName" value="Number of VMs"/>
</Value>
</Item>
</List>
</Property>
</Control>
<Control id="pagination-control_id_197" type="pagination-control" visible="true">
<Property name="start" value="0"/>
<Property name="size" value="50"/>
</Control>
</Controls>
<DataProviders>
<DataProvider dataType="list-view" id="list-view_id_194"/>
</DataProviders>
<Presentation type="list"/>
</ViewDef>
</Views>
</Content>
|
Warning:
You need to replace the Super Metric ID in Line 82!
You can gather your own Super Metric ID from the vRealize Operations Manager API using this PowerShell script:
1
2
|
$Response = Invoke-RestMethod -Method Get -uri "https://$vRopsFQDN/suite-api/>api/supermetrics/" -Credential $Cred
$Response.supermetrics.supermetric | Where-Object {$_.Name -eq "Datastore OverProv %"}
|
Create Alert
Alerts in vRealize Operations Manager are based on Symptom Definitions, so we need to create them first.
Symptom Definitions
I have created both Symptoms in two different levels (Warning and Critical), the third possible level is Immediate.
Number of VMs:
Overprovisioning Rate:
You can also import this XML file to create all Symptom Definitions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
<?xml version="1.0" encoding="UTF-8"?><alertContent>
<SymptomDefinitions>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-f2b3c5c6-2d99-46d1-917a-0939bf91a150" name="Datastore Overprovisioning-VMs Critical" resourceKind="Datastore" waitCycle="3">
<State severity="critical">
<Condition key="summary|total_number_vms" operator=">=" thresholdType="static" type="metric" value="150.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-0293538b-288d-47f7-beb0-389c1292ed85" name="Datastore Overprovisioning-Rate Critical" resourceKind="Datastore" waitCycle="3">
<State severity="critical">
<Condition key="Super Metric|sm_b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa" operator=">=" thresholdType="static" type="metric" value="200.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-e3d88bc6-f624-4433-9ce6-0154183d1529" name="Datastore Overprovisioning-Rate Warning" resourceKind="Datastore" waitCycle="3">
<State severity="warning">
<Condition key="Super Metric|sm_b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa" operator=">=" thresholdType="static" type="metric" value="150.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-f84e1cf7-9633-45ff-bb09-e43755fa2a2e" name="Datastore Overprovisioning-VMs Warning" resourceKind="Datastore" waitCycle="3">
<State severity="warning">
<Condition key="summary|total_number_vms" operator=">=" thresholdType="static" type="metric" value="100.0" valueType="numeric"/>
</State>
</SymptomDefinition>
</SymptomDefinitions>
</alertContent>
|
Warning:
You need to replace the Super Metric ID in Line 10 and 15!
You can gather your own Super Metric ID from the vRealize Operations Manager API using this PowerShell script:
1
2
|
$Response = Invoke-RestMethod -Method Get -uri "https://$vRopsFQDN/suite-api/api/supermetrics/" -Credential $Cred
$Response.supermetrics.supermetric | Where-Object {$_.Name -eq "Datastore OverProv %"}
|
Alert Definition
The new Alert is based on our Symptom Definitions and the Criticality of the Alert will be defined be the triggered Symptom Definition.
You can also import this XML file to create the new Alert Definition:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
<?xml version="1.0" encoding="UTF-8"?><alertContent>
<AlertDefinitions>
<AlertDefinition adapterKind="VMWARE" id="AlertDefinition-32273bbd-0cf0-489a-aea8-43a045bfb8ca" name="Datastore Overprovisoning-Rate" resourceKind="Datastore" subType="20" type="16">
<State severity="automatic">
<SymptomSet applyOn="self" operator="or">
<Symptom ref="SymptomDefinition-0293538b-288d-47f7-beb0-389c1292ed85"/>
<Symptom ref="SymptomDefinition-e3d88bc6-f624-4433-9ce6-0154183d1529"/>
<Symptom ref="SymptomDefinition-f2b3c5c6-2d99-46d1-917a-0939bf91a150"/>
<Symptom ref="SymptomDefinition-f84e1cf7-9633-45ff-bb09-e43755fa2a2e"/>
</SymptomSet>
<Impact key="health" type="badge"/>
<Recommendations>
<Recommendation priority="1" ref="Recommendation-df-VMWARE-StorageVMotionVM"/>
<Recommendation priority="2" ref="Recommendation-df-VMWARE-CheckDatastoreGrowth"/>
<Recommendation priority="3" ref="Recommendation-df-VMWARE-CheckUnexpectedVMProvisioning"/>
<Recommendation priority="4" ref="Recommendation-df-vCloud-AddMoreCapacityToDatastore"/>
</Recommendations>
</State>
</AlertDefinition>
</AlertDefinitions>
<SymptomDefinitions>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-f2b3c5c6-2d99-46d1-917a-0939bf91a150" name="Datastore Overprovisioning-VMs Critical" resourceKind="Datastore" waitCycle="3">
<State severity="critical">
<Condition key="summary|total_number_vms" operator=">=" thresholdType="static" type="metric" value="150.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-0293538b-288d-47f7-beb0-389c1292ed85" name="Datastore Overprovisioning-Rate Critical" resourceKind="Datastore" waitCycle="3">
<State severity="critical">
<Condition key="Super Metric|sm_b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa" operator=">=" thresholdType="static" type="metric" value="200.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-e3d88bc6-f624-4433-9ce6-0154183d1529" name="Datastore Overprovisioning-Rate Warning" resourceKind="Datastore" waitCycle="3">
<State severity="warning">
<Condition key="Super Metric|sm_b1ecbed3-6f1a-4fe4-8c7c-5bd4f36448aa" operator=">=" thresholdType="static" type="metric" value="150.0" valueType="numeric"/>
</State>
</SymptomDefinition>
<SymptomDefinition adapterKind="VMWARE" cancelCycle="3" id="SymptomDefinition-f84e1cf7-9633-45ff-bb09-e43755fa2a2e" name="Datastore Overprovisioning-VMs Warning" resourceKind="Datastore" waitCycle="3">
<State severity="warning">
<Condition key="summary|total_number_vms" operator=">=" thresholdType="static" type="metric" value="100.0" valueType="numeric"/>
</State>
</SymptomDefinition>
</SymptomDefinitions>
<Recommendations>
<Recommendation key="Recommendation-df-VMWARE-StorageVMotionVM">
<Description>Use Storage vMotion to migrate some virtual machines to a different datastore</Description>
</Recommendation>
<Recommendation key="Recommendation-df-VMWARE-CheckDatastoreGrowth">
<Description>Check if the datastore usage is a planned growth and expand the storage if necessary.</Description>
</Recommendation>
<Recommendation key="Recommendation-df-vCloud-AddMoreCapacityToDatastore">
<Description>Add more Capacity to Datastore</Description>
</Recommendation>
<Recommendation key="Recommendation-df-VMWARE-CheckUnexpectedVMProvisioning">
<Description>Check if there is an unexpected provisioning of virtual machines on this datastore.</Description>
</Recommendation>
</Recommendations>
</alertContent>
|
Create Email Notification
If you want to create a notification for the Datastore Overprovisioning Alert you need to configure a Trigger for the specific Alert Definition.
Possible Dashboard
To get a quick overview of your environment a Dashboard is great tool. My example shows the newly created View based on the selected vCenter. The Heatmap has also the vCenter from the Object List as Input and visualizes the Overprovisioning-Rate (size) and the Number of VM’s (color).
