When we measure on a CMM (Coordinate Measurement Machine) many points of a cylinder, what is the algoritm to find a cylinder center considering minimum cylinder circumscribed or a least square cylinder?
I am afraid that the equation Ahmed provided will work only if you have the set of points evenly distributed around the center (i.e. sampled at equally step angles) If so then it will provide the average circle (i.e. not the circumscribed nor Inscribed circle). In this case this is the easiest way to do it.
To solve your stated problem accurately you need to do more complicated calculations by generating sets of principle equations for each 3 points as we know that in Euclidean space, there is a unique circle passing through any given three non-collinear.
Perhaps the basic info provided in
en.wikipedia.org/wiki/Circumscribed_circle
may help you to understand the principles of this issue.
I have just forgot to add, in case of calculating cylindericity (cylinder is a 3D shape) rather than roundness (circle of 2D) then you need to extend the solution into 3D space rather than 2D (X-Y cartesian presentation).