Deploying enterprise applications using a tiered architecture is increasingly becoming a standard both in virtualized and non-virtualised environments. A major challenge in managing such applications is the ability to be able to quickly detect performance bottleneck/anomaly/surprises and identify which of the tiers is causing the perceived constriction.
The interest here is knowing what techniques have been used, or are being used for such tasks, especially in virtualised environment where applications are deployed in black-boxes (VM), and the infrastructure provider can only detect these changes and identify affected tier only by externally observing the performance of the applications.
In this case, how can one use external system level performance "vital signs" of virtual machines to be able to quickly detect anomalies, and identify affected tier?
For the open source Ceph project, the first line effort is to prevent bottlenecks in the first place. So data gets stored by a pseudo-random placement algorithm to prevent hot spots and load spikes. Computational placement also eliminates a centralized broker so applications can talk directly to the OSDs. Additionally, computational placement allows the cluster to rebalance dynamically. This document describes its architecture: http://ceph.com/docs/master/architecture/
Ceph block device volumes provide thin-provisioned images. One method to increase performance is to stripe the virtual image across the cluster: http://ceph.com/docs/master/architecture/#data-striping Again, this is something that gets done BEFORE problems arise.
To create VM images quickly, Ceph uses copy-on-write cloning of snapshots. http://ceph.com/docs/master/rbd/rbd-snapshot/#layering. Snapshot cloning is a technique that provides VM images to systems like OpenStack (http://ceph.com/docs/master/rbd/rbd-openstack/) and CloudStack (http://ceph.com/docs/master/rbd/rbd-cloudstack/). ProxMox integrates with Ceph in the same way.
The basic techniques for knowing if the data tier in Ceph is causing problems involves ensuring that the monitors and OSD daemons are running. A failure of multiple OSDs--like a rack switch failing--would cause some significant latency. To get a deeper idea of what's going on, a user will monitor placement group states. http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/
In large scale clusters, failures are an expectation not an exception. So the system needs to be built with that in mind.
How you can tell which part of the tier is failing without knowing what's happening in the underlying system is more problematic. Latency for a Ceph client could be the local software, a public network issue, the failure of some OSDs or monitors, rebalancing, the cluster reaching a near full capacity, or an issue with the cluster network (used in high scale clusters for faster replication and recovery).
Usually an infrastructure vendor at least publishes the basic status of the system. If they have an API, you can introspect into it that way. For example: http://dmsimard.com/2014/01/01/documentation-for-ceph-rest-api/ A dashboard with status can also be helpful.
Hmmm thanks John. I am going to look into the techniques offered by Ceph.
But I am afraid, isn't Ceph a Storage solution which would fit in the database tier of a cloud-based multi-tier application? As it is, our interests lie in the study of bottleneck detection in any of the tiers of a cloud service in an IaaS cloud.
Yes, the network infrastructure itself could be the bottleneck, I agree. But we are looking at the VM level, we want to be able to identify within a short space which VM is dragging performance and why. Well, do you have an idea about identifying network-induced bottlenecks?
- Badges
- Science method