VMWorld 2008 live: in depth Disk I/O monitoring
by Johan De Gelas on February 27, 2008 2:00 PM EST- Posted in
- Virtualization
In an absolutely brilliant and (of course!) purely technical session called "Disk Workload charactherization on ESX", Richard McDougall explained how you can solve disk I/O problems by capturing in depth monitoring data and analyzing it.
Simply type the comment below in the ESX Console (COS):
/usr/lib/vmware/bin/vscsiStats -h
and you can get a wealth of disk I/O stats. This vscsistat software seems to use a light weight probe that is part of the virtualized HBA drivers (Buslogic of LSI Logic). This of course implies that you are measuring at the level of one VM, you can not a complete overview of the usage of your storage arrays this way. It's purpose is to help you understand what the application running in a particular VM is doing. It also works if you use an iSCSI (or FC) iniator on the ESX hypervisor level as all blockcommands are still passing through the virtual HBA. If you use an iniator on your guest OS level, you won't get any stats however.
The following data can be gathered:
- I/O block size
- latency
- Seek distance/spatial locality
- Outstanding I/Os (queues)
- I/O arrival time/ Latency
In other words everything you need to know to get a very good idea about your application's disk access profile.
You can get histograms of latency, I/O blocksizes and so on. Unless you perform a full trace, Richard assured us that this is a very light weight process that can be run on production systems with very little performance loss.Knowing for example which blocksize your application uses most frequently can help a lot, as the blocksize of the guest should align with the blocksize of backend (your storage array).
Another tip I picked up:Transactions (commit) on databases consist of update writes on the logs (latency!), while most databases then perform async writes to the actual database. So you need to place your logs on a low latency array, while your disk arrray where your database data resides just needs to have enough bandwidth and write cache memory.
Facebook
Twitter
RSS
2 Comments
View All Comments
blckgrffn - Thursday, February 28, 2008 - link
Thank you - working through some of this right now, needed a good way to quantify storage needs other than capacity for our storage folks. Good stuff.Nat
JohanAnandtech - Friday, February 29, 2008 - link
You're welcome!The next days we'll have more on VMWorld and Virtualization in general.