Adsense Code

Thursday, 19 February 2015

Virtual Flash Read Cache - Fantastic - but something odd today... massive disk latency

We have been adding Flash Read Cache against selected VMDKs gently and we have been getting excellent results.  In the main, we have allocated against some of our virtual Citrix servers and we have noticed a decrease in the latency reported on the Citrix VMs and a reduction in the CPU utilisation on our SAN.

Our Netbackup Master server is also a virtual server and we thought with all those data reads, adding vFRC against the VMDKs on the 20% ratios that VMware recommend would only improve things.

Not so!

Over the morning, more and more VMs over different datastores, over different IO architectures, reported disk latencies.  We were, on a couple of VMs, getting disk latencies of over 22500 milliseconds (yes, 22 and a half seconds!).

It took a bit of digging and thinking of the only major disk IO change that we had done.  We removed the FRC against the master netbackup server and over about 10 minutes, everything started behaving properly again!

So - we have solved the problem - but puzzling on why that would be the case.  If anyone has any thoughts - do let me know!

No comments:

Post a Comment