I'd like to strongly recommend you take a look at our new software MIPAR at http://MIPAR.us. We have tools that would make it very easy to classify objects by convexity, either using moment invariants or convex area / area ratio. If you'd like to post an example image, I'd be happy to make a recipe file for you and show you what we can do. MIPAR is actually written in MATLAB, but runs entirely standalone without the user needing MATLAB installed. Finally, in MATLAB you may find help in using the regionprops function to measure an object's convexity, but this requires segmentation and labeling of the object, a task which MIPAR helps make very easy.

Cheers,

John

Hi Adeleh:

You can also test some convexity plugins for ImageJ.

All my best

Humberto

Similar question is addressed here, have a look on the following link :http://www.mathworks.com/matlabcentral/answers/56276-how-to-find-concavity-or-convexity-part-in-an-image

I recommend opencv library, there are many useful predefined functions for your case. 

http://docs.opencv.org/2.4/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html

Here's what you should do:

First of all, to identify the polygons you need to first transform the image to its binary mask. The Otsu threshold can be used for that:

Im = imread('your_image.tif');

otsu = graythresh(Im);

binary_mask = im2bw(Im,otsu);

In the binary mask you can find the 8-connected (or 4-connected) objects. For instance, you can do that using:

bwconncomp(binary_mask)

Once you have that done, you can easy find the edge pixels of each object (8-conneted pixels that have neighbors that are not part of the 8-connected objects), and run a line to all other edge pixels of the same 8-connected object. If some of these pixels are not part of the 8-connected object it is a convex.