Konstantinos,

I agree that most work on human activity has assumed pre-segmented data and addressed only the recognition part of the task. There are exceptions, however. To toot our own horn, at Colorado State University (as part of DARPAs Mind's Eye program) we tackle both detection and recognition together and as an unsupervised learning problem. It has been hard so far to get the whole system published (but see O'Hara & Draper CVPR 2012 & O'Hara & Draper WACV 2013 for pieces). The system is keyed by motion: moving objects are tracked, and a sliding window cuts video snippets from the tracks of moving objects. At training time, the snippets are clustered to form "activities". At performance time, activities in new tracks are labeled according to these activities. The goal is take a pool of unlabeled videos, learn what activities occur multiple times, and then be able to spot those activities in novel (unlabeled) videos.

Konstantinos,

You really got me thinking into possible ways to tackle this problem. One possibility is to explore possible solutions in other areas, and the one that comes to mind is speech recognition. If you take a look at speech recognition , it suffers from similar problems as action spotting, that is:

1) continuous domain: continuous audio continuous video

2) segmentation problem: phoneme identification object identification

3) transition identification: phoneme transition(especially with coarticulation) movement transition

4) noisy background : noise in recording(in the form of unwanted frequenciesnoise in video(in the form of artifacts, occlusions, etc.)

That immediately got me thinking about HMM's to model continuous movement as in speech recognition and as a matter of fact there has been very early and preliminary work in this area of HMM modeling. look at:

Gesture Spotting in Continuous Whole Body Action Sequences Using Discrete Hidden Markov Models by Park and Lee.

I would extend their work to continuous movement and also doing unsupervised learning of HMMs for action spotting. The idea is to let the unsupervised learning to try to fill in the gaps(take a look at Applying the Hidden Markov Model Methodology for Unsupervised Learning of Temporal Data by Li and Biswas)

In addition, I would also explore deep learning neural nets as alternative to HMM's to model actions. After messing around to see if they pan out in results, I would concentrate on obtaining a robust feature set that characterizes the actions(since the models that the HMMs build are dependent on the features extracted).

My two cents.

Hope this helps

Dear Arturo, all.

Thank you for your helpful guidelines. I read the paper carefully and I should admit that it's very close on what I'm searching for. However, as more I think action detection problem so much the deficits that I encounter.

Conceptually, the problem constitutes of two-three main challenges:

1) The first one focus on continuous motion actions. In this occasion, it's very difficult to find real-time start and end points of each action sequence. Some researchers choose to work with fixed temporal windows and online classification (using SVM):

http://hal.inria.fr/docs/00/51/48/45/PDF/paper.pdf

2) Another challenge, but considered to be solved, is when we have interrupted video frames, which can be detected by a shot detection alg.

3) Furthermore, still images (or static video frames) within a video sample may contain a human activity. However, they are usually neglected as they are motionless and human activity recognition algorithms usually made the assumption that they are regions out of interest.

I think, there is a lot of work to be done on this topic.

Regards

Konstantinos

Konstantinos,

I agree that most work on human activity has assumed pre-segmented data and addressed only the recognition part of the task. There are exceptions, however. To toot our own horn, at Colorado State University (as part of DARPAs Mind's Eye program) we tackle both detection and recognition together and as an unsupervised learning problem. It has been hard so far to get the whole system published (but see O'Hara & Draper CVPR 2012 & O'Hara & Draper WACV 2013 for pieces). The system is keyed by motion: moving objects are tracked, and a sliding window cuts video snippets from the tracks of moving objects. At training time, the snippets are clustered to form "activities". At performance time, activities in new tracks are labeled according to these activities. The goal is take a pool of unlabeled videos, learn what activities occur multiple times, and then be able to spot those activities in novel (unlabeled) videos.

Hi, I am aware of several approaches in this area. we can discuss it if you want

I am aware of several approaches. Let us discuss if you want

Would you like to post them on this thread, so that everyone can have them on their disposal? It would be fine if you have some proposals and appropriate links (i.e. published papers) to share with us ? I really appreciate for your time

Well, i have a rather short comment here:

Wearable computing. A great deal of the tasks wearable computers should be assisting the user in require the computer to recognize a task the human is engaged in.

The TZI of the University of Bremen worked for years in the field of wearable computing ( Wireless Winspect, Scipio module, KATO , Chronious , ... ).

However as a wearable computer is unable to observe its humans activities from an external point of view, as, for example, a helper dog can, we didn't do visual recognition, but used sensors at strategic locations and then machine learning algorithms to make the computer understand what the user is doing.

On average, the resulting trained systems had a high rate of recognizing a task being done eve nwhen a different human was doing it.

However such an approach needs a totally different setup than what you have. You'd need sensorpacks instead of cameras, and I .. somehow... have the feeling that you don't plan on starting all over with a different technology.

Anyhow, just mentioning it as so far nobody spoke about that way of detecting human activity.

Konstantinos,

Human activity has different level of complexities (simple movements, context based movements, etc.). if your focus is on simple one you can use boundary detection-based approaches to segments different actions through a sequence of human activities.

In complex ones, as Aggarwal&Ryoo-2011 in ( "Human activity analysis: A review" ) were concluded; “hierarchical approaches have their advantages in recognition of high-level activities performed by multiple persons, and they must be explored further in the future to support demands from surveillance systems and other applications”.

You can see a sample of hierarchical approach to estimate different level of human activities on our recent work "Probabilistic human interaction understanding: Exploring relationship between human body motion and the environmental context". For more information you can check my other publications.

For your other concerns about human activities segmentation ( real-time start and end points of each action ), which was ours also. Weinland et al.2011's survey can be valuable for you. As I mentioned for the simple actions which the boundaries of the action transitions are possible to be defined from input signals, boundary detection approaches can solve the problem.

In the complex situation, the boundary detection-based approaches can not be applied easily, and usually sliding window-based approaches are used. which usually can not detect exact start and end point of each actions. which is our ongoing work to find a solution for that.

Hi you could check this paper:

http://www.robots.ox.ac.uk/~minhhoai/papers/MMED_CVPR12.pdf

It's a bit tangential to your problem, but you could look at my group's work on lie detection which uses non-verbal microgestures accumulated over timeslots you can find downloadable papers at:

http://semanticsimilarity.net/list-of-publications/

look for

Silent talker: a new computer-based system for the analysis of facial cues to deception

and

Charting the behavioural state of a person using a Backpropagation Neural Network

Pl. look at this paper - "Eigenbehaviors: identifying structure in routine" - Nathan Eagle & Alex Sandy Pentland, Behav Ecol Sociobiol (2009) 63:1057–1066

DOI 10.1007/s00265-009-0739-0, Published online: 7 April 2009

# Springer-Verlag 2009

Smart phones can be used to detect such context - "A survey on smartphone-based systems for opportunistic user context recognition", Hosseini et. al.,

ACM Computing Surveys (CSUR) Surveys, Volume 45 Issue 3, June 2013

Article No. 27

http://dl.acm.org/citation.cfm?doid=2480741.2480744

Volume 45 Issue 3, June 2013

Article No. 27

Or you can check my paper on active event recognition. There we face the problem of moving the camera to understand what an human is doing. We are doing some new work trying to recognize several events at the same time with a camera, like humans do moving their eyes. If you consider that once you have detected a human you can assume that he is doing an action (even some null action), the problem of detecting an action becomes that of detecting an human and recognising online his actions.

Let me know what you think

Not strictly related to human activity, rather Transportation mode, but you might want to take a look at our paper.

https://www.researchgate.net/publication/236883633_By_train_or_by_car_Detecting_the_user's_motion_type_through_smartphone_sensors_data?ev=prf_pub

Conference Paper By train or by car? Detecting the user's motion type through...

As Bruce mentioned above this is a focal point in the DARPA Mind's Eye program. All teams worked hard on that problem last year -- and it turned out to be hard indeed.

The approach of the TNO CORTEX team here is to run an instanteneous action classifier (based on STIP + RF + SVM) at every second for every track. This provides the option to detect the action without need for a temporal segmentation. More info is at: http://home.kpn.nl/henri.bouma/research/Burghouts_2013_ICPRAM_SearchEngine48Actions.pdf

We worked on a wearable device (currently a tiny yellow box to attach on your belt or ankle) who recognizes "daily living activities" like "walk, stand, sitting, ..."

http://groupware.les.inf.puc-rio.br/har

We also investigated "how well" (instead of "what") an activity is performed in the context of Weigth Lifting Exercises. You may also find this article following the link below.

I supervised a graduation project about " Gesture Control"

Replacing the computer mouse by tracking the motion of a human fingers.

This process includes:

1- preprocessing (filtering for all channels (RGB))

2- segmentation using a simple thresholds

3-finger tip detection

This framework was implemmented using MATLAB software and a webcam used for detection the gesture.

I would like to thank everyone for your helpful input.

After some study that I have done in (computer vision) SoA, I gathered some useful material that I would like to share with you:

1) http://www.irisa.fr/vista/Papers/2007_iccv_laptev.pdf

2) http://lear.inrialpes.fr/pubs/2010/KMSZ10/KlaserMarszalekSchmidZisserman-SGA10-ActionLocalization.pdf

3) http://class.inrialpes.fr/pub/willems-bmvc09.pdf

4) http://www.cs.cmu.edu/~yke/video/iccv2007.pdf

5) http://www.ifp.illinois.edu/~cao4/papers/yuxiao-MIAction-ICCV09.pdf

6) http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.9650&rep=rep1&type=pdf

7) http://lear.inrialpes.fr/people/triggs/events/iccv03/cdrom/iccv03/0726_efros.pdf

Hope you would find them as useful as I did.

Regards,

Konstantinos

Most of the work in literature is on action recognition and assumed pre-segmented data. Many datasets have been evolved in this regard. In most of these datasets, the pre-segmented actions are provided and are performed by one or more subjects. The following link contains the information of the datasets for action recognition task.

http://naveenideas.blogspot.com/2017/02/datasets-action-recognition.html

http://www.di.ens.fr/willow/events/cvml2010/materials/INRIA_summer_school_2010_Ivan.pdf

http://users.eecs.northwestern.edu/~mya671/mypapers/ICCVW09_Yang_Lv_Xu_Yu_Gong.pdf

https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Yu_Fast_Action_Proposals_2015_CVPR_paper.pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.4053&rep=rep1&type=pdf

https://openreview.net/pdf?id=HJfDVHkag