first of all thank you very for excellent question in the discussion,

According to my opinion the use of gesture detection sensors in healthcare is good idea and practices. However, this is a new technology therefore it required some time to build in practices and also it required some good level of expertise.

According to my second suggestion, as we know Microsoft Kinect mainly build for gaming propose, however the use of Kinect can be much useful in stroke rehabilitation via gaming. Using the kinect game we can able to provide natural user interface in which results the patients can engage, motivate and can learn more, as compare to traditional stroke therapy.    

However, the use of Kinect in healthcare required safety i.e. during the patient play, patient should not play fast  and not to take much stress otherwise it can be more serious.      

In the last decade, many researchers have developed different type of devices which were able to manage and recognize motions of hands. Two years ago, Leap Motion showed its device which is able to detect hand, fingers, part of hands (e.g., palms) with high precision. After this demonstration, Microsoft decided to improve the quality of kinect in order to also add fingers detection.

Thus, Kinect (the last release) and Leap Motion device are actually the best device to interact with applications that have a professional final purpose, such as industrial and healthcare fields.

Furthermore, the real issues about gestures detection are the following:

I hope to give you a good summary of my personal point of view about the use of gesture recognition.

Regards,

Andrea

The main issue with the Kinect in specific is still the limited area of gesture recognition. Thus the device has to be relatively close to the surgeon. It may be attached to ceiling mounted monitor setups, but still, as the Kinect is not certified for this use, the active air ventilation of the device is a concern as the device itself can pollute steril areas.

Therefore I'm skeptical that you'll find the Kinect in its current revisions directly in the OR, even when it is placed in the semi-steril areas. There has to be a modified version of it to match the prerequisites. But actually, I'm not aware of similar devices which are designed that way to meet these specific requirements within OR setups.

The current practise in the OR is more or less, if the surgeon has the intention to request different medical images, that an assistant operates the interfaces for him. Or that sterility is achieved via steril enclosures of the haptical interaction devices and an extra pair of steril gloves before the interaction and afterwards, if the surgeon uses devices directly. This is certainly an limitation and not really a best practise, but it is not uncommon that the surgeon wears more than on pair of gloves on his hands. It's more or less a pragmatical workaround and not every surgeon is willing to do it that way. 

These limitation could be avoided by hand recognition techniques for sure and might also work with the surgeons gloves which are contaminated. The gestures might be a bit more complex to cover the whole spectrum of interaction with DICOM datasets.

In general, the Kinect technology is relatively new and will be refined in future. The average OR is still not equipped with monitor set-ups. the available medical images / datasets are primarily consulted and studied prior to the surgical intervention. The availibilty of these data during the interentions could be beneficial, but still, these data are still an approximation of the patients condition as they are retrieved several days or even weeks before. Thus the scope is in gerneral limited and the surgeon has to use different exploration techniques by physical inspection or by the use of ultrasound monitoring to gain an insight of the current patients condition.

I personally do not think Kinect is going to be suitable for a surgery scenario, but if you wish to use Kinect to demonstrate the potential of future devices and how different scenarios can already be achieved, I would say it is still feasible. However, for a longer term research direction, it might be more worth it to set up your own HD cameras and construct own infrastructure to show future scenarios.

why not with some wearables? if you are only interested in hand-based gestures, perhaps even wearable devices will be doable (gloves with IMU sensors?)

Well, I am current doing some research with Kinect Sensor but mainly for full body gesture tracking and recognition. Kinect itself has some limitation especially the visible range. That's mean you need to stand / perform hand gesture within certain area. The Kinect may not be able to detect and recognize a gesture precisely if the subject is out of range. 

I am kind of agree with Sian Lun Lau's suggestion to use wearable device if you ever consider to scrolling medical images while moving around in the surgeons room.

I think Kinect could well be used to let the surgeon scroll through a volume, pan, zoom and adjust brightness/contrast. For selecting another scan/modality with a dicom-interface... probably speech recognition is a better option.

in a pure technology enabled scenario, any 3d doppler/ultrasound/xray/tomography  scan can be projected right over the patient's body, providing an Augmented Reality overlay which may be thus controlled via KINECT's gesture librarys and basically, have a ready reference of the patients scans (taken earlier) during or before operating providing seamless data and minimal multitasking and distraction.

@Andrea Vitali: Thank you for your valuable answer. I have seen demos about Leap Motion technology but I do not think its suitable for OR. However, Kinect is better choice. I have tried Kinect myself but I think it has some limitations which need to be improved in the future. GestSure company is trying to adopt Kinect 2 which is more reliable to be used in OR.

btw, I am eager to know about your personal experience with gesture recognition. ^_^   

@Jan Wieferich: Thank you for your helpful information. I am totally aware of Kinect limitations but it it still the most popular tool for gesture detection and recognition. Kinect 2 is coming soon to the markets and I hope Microsoft could eliminate some of the limitations of Kinect 1. 

@Sian Lun Lau: Kinect in its current features are still not suitable for OR. However, Kinect 2 is going to be better and more reliable (as Microsoft claimed). Regarding gloves with IMU sensors, I do not think they are going to be much of comfort with surgeons since they might distract them while doing surgery and gloves might compromise the sterile procedures as well. 

@Lambert Zijp: I personally did the voice recognition experiment and it was somehow successful to control my 3D objects. Therefore, your suggestion, in my opinion, is the best to be used in OR to control the views and go through other modalities. Since Kinect has both gesture detection and voice recognition techniques, I believe it is going to be one of the top candidates for OR environments. 

As far as I know (my brother is surgeon) they have some devices controlled by using feet (e.g., live X-Ray when they are reallocating a bone). Using feet usually do not present problems in sterile environments.

In case you have some limitations with gestures, maybe you could try using simple controls (e.g., right and left, and long press for enter and back) by means of some "foot buttons". Then, obviously, using a proper user interface.

My 2 cents.

Hi all

Our team has already experience with gesture detection in forensic radiology.

Have a look at these papers if you are interested in this topic:

https://www.researchgate.net/publication/262768309_Out_of_touch__A_plugin_for_controlling_OsiriX_with_gestures_using_the_leap_controller?ev=prf_pub

https://www.researchgate.net/publication/51782034_You_Can%27t_Touch_This_Touch-free_Navigation_Through_Radiological_Images

https://www.researchgate.net/publication/237870899_Invisible_touchControl_of_a_DICOM_viewer_with_finger_gestures_using_the_Kinect_depth_camera?ev=prf_pub

Article Out of touch – A plugin for controlling OsiriX with gestures...

Article You Can't Touch This: Touch-free Navigation Through Radiolog...

Article Invisible touch—Control of a DICOM viewer with finger gestur...

In my opinion Kinect is just the beginning of unconstrained gestural interaction. Precision might not yet be really suitable for unambiguous interaction, but as it has been said, new 3D sensors are coming. However, mulitmodal interaction should gain more interest, as gesture combined with voice offers less ambiguity, but also allows to control an interface with let's say your voice, while your hands are busy.

Even if gestural interaction is only used for browsing through images, and not actually for the operation itself, it should be as reliable as possible so that the surgeon does not waste time or starts being nervous because a gesture would not be recognized by the system. For this reason, research still needs to be made, to provide more reliable interaction. Amongst others, multimodality is probably an answer to that problem (gaze tracking, voice recognition, ...).

This  emerging  technology  is  a  plus  to  the  OR  but  I  foresee  some  drawbacks.

I carry out some procedures in the theatre as a Radiographer and I don't think the hand is free enough for gesticulations unlike the leg.The hand  will  be busy  during  surgery  and  will  need  to  pause  to  carry  out  the  gestures. Voice  may  be  fair  enough  but  I  want  to   think that  a  seemingly  'idle'  organ  like  the  leg  should  be  the  target  of  the  manipulations.

It  only  needs  the  eye  to  see  that  the  leg  is  manipulating the  right  control, that's  all. With  that  the  hand  remains  busy  while  still  reviewing  images.

But  if  we  are  talking  about  review  after  surgery  then  this is  a  great  feat  with  a  lot  of  promise.

I am not really convinced yet because of the following reasons:

First, the resolution of the sensors is still not good enough.

Second, since every user performs a gesture in a different way, it needs sophisticated filtering and interpretation algorithms, which do not exist yet.

Third: Personnel has to learn gestures, in particular if a larger set of commands should be entered.