Comparing the clinical performance of ultrasound scanners in an objective way is very difficult.  We carried out a project where a group of 8 sonographers ranked 11 scanners of different ages and purchase costs; their rankings showed good agreement with each other.  We then ranked the scanners using our automated QA results: resolution, penetration, contrast etc.  There was very poor agreement between our QA based rankings and the sonographers' rankings.  One of the difficulties here is that, on modern scanners, there are so many possible combinations of settings available and some fancy processing that can make images look very different.  This makes it very difficult to do a fair like-for-like comparison.  A question that has been difficult to answeris "what makes a good clinical image?".  If clinicians can answer this question in terms that scientists can turn into a test we may be able to make some progress.

No, no existe tal escala. Los equipos de ultrasonido diagnóstico han evolucionado con el desarrollo científico técnico en busca del logro de una imagen más nítida que permita diagnósticos más certeros. La diferencia en la calidad de las imágenes ultrasonográficas obtenidas en diferentes equipos es el reflejo de esa evolución. Algunos diagnósticos pueden ser realizados sin lugar a dudas en equipos de diferente generación; en este grupo pudieran mecionarse la presencia de líquido libre en cavidad abdominal; quistes hepáticos, renales, mesentéricos o litiasis o calcificaciones mayores de 5 mm en cualquier localización al alcance de la técnica ultrasonográfica; también pudieran incluirse la mayoría de los nódulos o masas en órganos intrabdominales, miembros o glándula tiroides, por poner solo algunos ejemplos. Existe una limitación real en la aplicación del ultrasonido diagnóstico y es precisamente la variación entre los observadores; la calidad del diagnóstico ultrasonográfico dependerá entonces no solamente de la calidad de la imagen obtenida por un equipo sino también de los planos que se busquen para lograrla, de la expecticia profesional de quien manipula el equipo  y de la interpretación que el profesional realice de la imagen que obtiene. Ciertamente, los equipos de menor resolución tiene un margen de error superior a los esperados que cometa un profesional que trabaje con un equipo tecnológicamente superior. He visto las dos cosas: diagnósticos brillantes en equipos tecnológicamente más atrasados y diagnósticos no tan precisos en equipos tecnológicamente más avanzados.

A lot of image-producing machines do a lot of "PhotoShopping" before presenting the images to the user.

it might be an idea to see how the machine is comparet to itself, before comparing to others.

what we did on 3d-cbct scanners was:

make one scan, to make sure the scanner is "warmed" up

wait for 30 minutes, take 10 scans with 30 minutes between.

wait for 15 minutes, take 10 images with 15 minutes between.

after that you save the images, converts them to numbers, and then subtract.

you calculate for each subtraction, how many image elements, that was actually reproduced.

That (expressed as %) would be a fair expression for quality.

(for dental 6 different dental cbct-scanners the ability to reproduce was 2-15%) 

There are some objective image quality measures which may be used to quantitatively measure image quality. Additionally there are some methods which may be used to overcome difficulties arising when acquiring images from different scanners and different operators. Pease take a look in some recent publications made by our group with accompanied software in executable form which can be downloaded from research gate.

C.P. Loizou, C. Theofanous, M. Pantziaris, T. Kasparis, “Despeckle filtering software toolbox for ultrasound imaging of the common carotid artery”, Comput. Meth. & Progr. Biomed., vol. 114, no. 1, pp. 109-124, 2014. Best paper, editor’s choice

C.P. Loizou, M. Pantziaris, C.S. Pattichis, I. Seimenis, “Brain MRI Image normalization in texture analysis of multiple sclerosis”, J. Biomed. Graph. & Comput., vol. 3, no.1, pp. 20-34, 2013

C.P. Loizou, C.S. Pattichis, M. Pantziaris, T. Tyllis, A. Nicolaides, “Quality evaluation of ultrasound imaging in the carotid artery based on normalization and speckle reduction filtering,” Med. Biol. Eng. Comput.,” vol. 44, no. 5, pp. 414-426, 2006

C.P. Loizou, C.S. Pattichis, “Despeckle filtering algorithms and software for ultrasound imaging,” Synthesis Lectures on Algorithms and Software for Engineering, Ed. Morgan & Claypool Publishers, 1537 Fourth Street, Suite 228, San Rafael, CA 94901 USA, June 2008, ISBN-13: 9781598296204.

Thank you for your valuable in put. Mr C P Loizou you have done nice work. Thank you for responding. Till date there is no minimum level of performance which may be considered must for an usg machine to offer reasonable image quality. I have to perform ultrasounds on very basic usg machines ( some times unable to show the boundaries of skull of the fetus the very basic in antenatal fetal biometry !) in a rural set up in my country. The image quality was too poor to comment any thing about the disease in question. This is why I am searching some standard scale on which different. Machines to be compared and devise guidelines  stating a minimum performance level of an ultrasonography machines.

Dear Sandeep, Definitively, when comparing image quality, different factors should be taken into consideration and then combined together in order to have a final result. Image quality evaluation is a difficult process. The direction anyway is to compare always the performance of the quality metrics with the decision of the observers which is considered to be the ground truth. Good luck with your research

Dear Sandeep.  Your last post is interesting.  I've always been reticent about using objective test object measurements to determine suitability of scanners.  I encourage users to perform a clinical evaluation to check that the scanner meets their expectations; I think this may be what Christos means when he refers to "the decision of the observers, which is the ground truth".  The minimum standard for a scanner is that it is capable of performing the desired task in most cases.  To use your example, a scanner must be capable of displaying the fetal skull to sufficient quaity to allow accurate measurement.  In other fields the requirements might be, for example, to show hydronephrosis or to facilitate measurement of the carotid intima-media thickness.  I think that, until we have objective measurements that agree with the decision of expert observers, we should rely on users to make the judgment whether a scanner is fit for purpose.  I believe that we can make objective measurements to show whether the performance of a scanner has changed (QA) but not to show whether a scanner is fit for purpose.

@ Nicholas and Christos ( with regards) : I was hesitant to put forth this query of mine but since it was troubling me so i wrote it here.

I observed that since radiologists graduate from different medical schools there lies gross difference in what they 'perceive' is 'perfect' and secondly the administrators are not radiologists ( and sadly they are to decide while purchasing a machine by any purchase committee). That is why I insisted on an objective criteria. Another point is to define a minimum level of performance ( not worried about what the best is offering after too much of 'algorithms stuff') , as Nicholas mentioned, should be able to do majority of the work up to a acceptable level. Our journals talk of major and big researches (and they should, for the radiology to grow) but such problems which a fresh radiologist faces while practicing radiology at grass root level with the machines which we only see ( if lucky enough) in museums, should be addressed some where. The point is to attain a 'horizontal growth' delivering some basic radiology services to all people. 

This is a good idea to evolve some guidelines stating that 'this should be seen ( fetal skull tables for examples) but till now there are guidelines  or a list of things for radiologist to look for in specific scan ( e.g. ACR guidelines for antenatal usg) but no guidelines for performance of machine. The reasons are obvious- they do not face such problems. And if we can reach a consensus for that 'bare minimum acceptable level' it will be easier to have good quality machines in any future installations or at least retire the out dated systems.

Dear Sandeep and Nicholas,

Your comments are very valid. The direction on image quality evaluation is to find objective image quality criteria for evaluating the performance of the algorithms (or the machines). The problem is that the ground truth image is usually not available. This is always the case in ultrasound or MRI examinations. Nicholas can maybe also verify that, who believe practices clinical evaluation of images. That is why some researchers work also with Phantoms. These are artificial images of known characteristics that resemble a specific tissue and can represent a ground truth image. After processing of the phantom/s with an algorithm these characteristics are again evaluated either objectively or subjectively (by using metrics or by visual inspection). Thus the performance of the algorithm or system may be better evaluated. This “minimum acceptable level” as you pointed out is very difficult to be found objectively. Extensive experimentation with a large number of observers and a large sample of images is required for this task.  Nevertheless, I believe that the use of phantoms should be further utilized and used for the correct image quality evaluation process. Examples for the use of phantoms are given in my Book (C.P Loizou & C.S Pattichis, 2008). I am also preparing a new book, which will appear middle of next year with extensive experimentation on ultrasound phantoms using quality measures and multiple observers’ evaluation.

Thank you Christos sir for replying with so much detail and enthusiasm. It good idea to standardize the usg machines using phantoms.Doing so it will be easy to establish the minimum level ( and shall problems like mine) but it will be difficult to measure/calibrate or compare the best performance of different machines. In that case you may need to compare Usg against more definitive modality like MRI. Still comparing 'test images' of usg machines against MRI images. that too objectively will require some serious mental work. Very best for you upcoming book. Thanks once again.

I had an interesting problem a couple of years ago, where users liked one scanner and not another (of the same model).  When I looked at the scanners the default presets were different, so I redefined the preset on the disliked scanner to match the preset on the liked scanner.  The users were happy with this change.  At the same time I tested the scanners using a phantom.  With the new preset the resolution and contrast of the 2 scanners matched very well, but the grey-scale contrast and the visibility of anechoic targets with the new preset was reduced compared to the original preset.  Christos, I'll be very interested to see your work, as we do need to make some progress in this area.

hi Sandeep.

In my field (rheumatology) I ran in to similar problems. In rheumatology the most important modality is Power Doppler. We have two US machines in our center and I had the feeling I saw PD signal on one machine where it was absent on the other. So we built a power Doppler flow phantom, this was recently published (Arthritis Research & Therapy "Very different performance of the power Doppler modalities of several ultrasound machines ascertained by a microvessel flow phantom"  --  it is open acces so you should be able to get a hold of it, otherwise send me an email).

it appears however you are only interested in greyscale US is it not? Then too the abovementioned paper may help you. You write you want a minimum performance of greyscale imaging, it should at least display the fetal skull properly. You can find in the article how you make tissue mimicking material. That is easy. You can make a layer of tissue mimicking material as thick as an average cutis, subcutis, uterus etc of women in your area and place a bone on the bottom of  the container. This way you can test several US machines on the same phantom, checking whether the minimum criteria that you chose (detecting bone at a certain depth) apply to which machine.

Good luck

David

Thank you David for replying. I discussed the grayscale images as I am trying to a minimum acceptable level, PD is a step ahead. Its a nice suggestion to make a indigenous phantom and compare. I shall surely look into it. Thanks for sharing your article.

Hi Sandeep,

You could try QA4US, http://music.runmc-radiology.nl/index.php/MUSIC_qa4us.

//Andrew

Thank you Mr Andrew

You could refer to this work:

http://www.sciencedirect.com/science/article/pii/S0301562903011360

Thanks @ Mark Brewin

Buy an ultrasound phantom and test Your scanners using the same settings and make up Your own mind of what you prefer.

Intuitively, an imaging system,s ability to detect objects of low contrast against the background is determined by the following three factors : contrast (i.e., mean brightness difference between the object and its surrounding background), size of the object and noise level.

Article Image quality evaluation of ultrasound imaging systems: adva...