Deploying enterprise applications using a tiered architecture is increasingly becoming a standard both in virtualized and non-virtualised environments. A major challenge in managing such applications is the ability to be able to quickly detect performance bottleneck/anomaly/surprises and identify which of the tiers is causing the perceived constriction.
The interest here is knowing what techniques have been used, or are being used for such tasks, especially in virtualised environment where applications are deployed in black-boxes (VM), and the infrastructure provider can only detect these changes and identify affected tier only by externally observing the performance of the applications.
In this case, how can one use external system level performance "vital signs" of virtual machines to be able to quickly detect anomalies, and identify affected tier?
For the open source Ceph project, the first line effort is to prevent bottlenecks in the first place. So data gets stored by a pseudo-random placement algorithm to prevent hot spots and load spikes. Computational placement also eliminates a centralized broker so applications can talk directly to the OSDs. Additionally, computational placement allows the cluster to rebalance dynamically. This document describes its architecture: http://ceph.com/docs/master/architecture/
Ceph block device volumes provide thin-provisioned images. One method to increase performance is to stripe the virtual image across the cluster: http://ceph.com/docs/master/architecture/#data-striping Again, this is something that gets done BEFORE problems arise.
To create VM images quickly, Ceph uses copy-on-write cloning of snapshots. http://ceph.com/docs/master/rbd/rbd-snapshot/#layering. Snapshot cloning is a technique that provides VM images to systems like OpenStack (http://ceph.com/docs/master/rbd/rbd-openstack/) and CloudStack (http://ceph.com/docs/master/rbd/rbd-cloudstack/). ProxMox integrates with Ceph in the same way.
The basic techniques for knowing if the data tier in Ceph is causing problems involves ensuring that the monitors and OSD daemons are running. A failure of multiple OSDs--like a rack switch failing--would cause some significant latency. To get a deeper idea of what's going on, a user will monitor placement group states. http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/
In large scale clusters, failures are an expectation not an exception. So the system needs to be built with that in mind.
How you can tell which part of the tier is failing without knowing what's happening in the underlying system is more problematic. Latency for a Ceph client could be the local software, a public network issue, the failure of some OSDs or monitors, rebalancing, the cluster reaching a near full capacity, or an issue with the cluster network (used in high scale clusters for faster replication and recovery).
Usually an infrastructure vendor at least publishes the basic status of the system. If they have an API, you can introspect into it that way. For example: http://dmsimard.com/2014/01/01/documentation-for-ceph-rest-api/ A dashboard with status can also be helpful.
Hmmm thanks John. I am going to look into the techniques offered by Ceph.
But I am afraid, isn't Ceph a Storage solution which would fit in the database tier of a cloud-based multi-tier application? As it is, our interests lie in the study of bottleneck detection in any of the tiers of a cloud service in an IaaS cloud.
Yes, the network infrastructure itself could be the bottleneck, I agree. But we are looking at the VM level, we want to be able to identify within a short space which VM is dragging performance and why. Well, do you have an idea about identifying network-induced bottlenecks?
Let's make an example for better discussion.
If it's a 3-tier architecture, the first tier consists of VMs dealing with client requests. All requests are then pushed into a queue like SQS or rabbitmq and then processed by the 3rd tier (scalable VMs).
You can measure the average response time (ART) of the first tier. If the first tier is working fine, ART should be evenly distributed over time. If it's rising, add more VMs to the first tier (or fix the reverse-proxy) and keep measuring.
For the second tier (the queue), you can measure time in queue (TIQ). If it goes up, add more VMs at the 3rd tier.
If you can't touch any of these components (really black-boxed),
one way is to randomly generate clients all over the world and measure ART and see if it's unbalanced.
- Badges
- Science method