Hi Victoria. I do not really agree with the previous comments / answers. Data Cleaning has 2 parts. The first one and often the most time consuming is checking that all values (quantitative or categorical) are as expected (e.g. Format, do you allow NULL / NAN, ...). For this I will use a dedicated perl code for example. And then (second part) once you are sure about the content / values you can use statistics. BUT remember that "outlier removal" first is a question of where you put the threshold AND second and even more important this should always be justified / confirmed by experimentators. Variability is inherent to any experimental data. Never remove outliers if you cannot understand why they are outliers. Last but not least: Statisticians / Biostaticians use often an arbitrary threshold of 95%. This is purely arbitrary decision! And often i is possible that this leads to under/over fitting. If you want some help, just have a contact [email protected].

Thank you for your quick response and the citations. I can certainly see a way to weed out outliers, anomalies, perhaps omissions. Do these data mining methods mostly work for quantitative data? What if it's mostly qualitative textual data (e.g., blogs, forum posts, RSS feeds, etc.)? Thanks again!

Hi Victoria. I do not really agree with the previous comments / answers. Data Cleaning has 2 parts. The first one and often the most time consuming is checking that all values (quantitative or categorical) are as expected (e.g. Format, do you allow NULL / NAN, ...). For this I will use a dedicated perl code for example. And then (second part) once you are sure about the content / values you can use statistics. BUT remember that "outlier removal" first is a question of where you put the threshold AND second and even more important this should always be justified / confirmed by experimentators. Variability is inherent to any experimental data. Never remove outliers if you cannot understand why they are outliers. Last but not least: Statisticians / Biostaticians use often an arbitrary threshold of 95%. This is purely arbitrary decision! And often i is possible that this leads to under/over fitting. If you want some help, just have a contact [email protected].

Hi Victoria. I agree with Pierre Lecocq in that the first step is cleaning up the data to fit the expectations of your statistical software (e.g. adding coding for missing values, removing boilerplate code if present, etc). Since my background is in corpus linguistics, I consider this the first step anyway, big or small data (corpus data can often be messy no matter the size of the corpus).

When it comes to outliers in linguistic data, specifically word frequency distributions, one solution could be to look for high probability words that are overly influential, and perhaps apply some sort of transformation. Other exploratory steps could also be useful, depending on exactly what you're planning to do and what you're looking at in the textual data (words, phrases, sentences, paragraphs?). Harald Baayen's excellent book "Analyzing linguistic data: a practical introduction to statistics using R" (CUP, 2008) has some useful discussion of outliers and linguistic data.

Thank you for the advice. I was just looking into R packages for NLP. I will dig up that book you recommend. I am open to further solutions, if you (or anyone else) can think of anything else useful. Thanks.

Hi Victoria. Just one question (it's curiosity): What exactly are your data?

Our datasets primarily consist of texts (un- or semi-structured ones) of the type you're likely to see in sentiment analysis (personal stories from blogs, reviews, news RSS, forum posts, tweets, etc). We're working on the deception detection task, currently looking into the news discourse/ genre (authentic amusing, humorous, implausible vs. fake ones). We often start "small" to model the problem and scale up (down up), but I'm particularly curious about how this could be modeled, so to say, top down (from larger patterns to individual instances, like in fraud detection). Does this help with the context? Thanks for the suggestions by the way. I do agree outliers are worth manual error analysis or at least spot-checking. In fact, in fraud and spam detection, those are the gold nuggets, aren't they? As long as you know how to dig them up...

Another question is what should be considered an outlier. At least within regression modeling we can sort of consider outliers as data points (or clusters of points) not fitting the model. But where does the problem lie? Is it with the data or the model? Aberrations in the data might be caused by data entry errors, data encoding problems, extreme results suggesting non-compliance with a task (in an experimental setting), or something going pear shaped with a Perl or Python script pre-processing the data. However, if all those sources undesired variation can be ruled out (plus other similar ones I might have missed), then it might be useful to think of extreme values not as outliers but as a naturally occurring property of the data (as Pierre Lecocq also suggests). In that case, my intuition would be to change the model. I think that outlier detection and removal should not be considered some kind of mechanic procedure, it must be done with a careful consideration of the data in mind.

The problem here probably lies in the fact that you have to establish two models, one for ordinary text and another for fake text. Your program should decide which category a new text fits in.

My next suggestion is that your corpus might be split into different types, tweets are very different from blog stories. You should have confidence in your text classifier to classify texts according to their type before you attempt to tag them as true or fake.

I fully agree with Pierre Lecocq and Gard Jenset. Data is what you work on, as etymology says, it is given. So data should not be described from the start as "having errors". Errors are all in the processing or in the model. In the same way, when describing text as not structured, one expresses decision to discard structure.

I agree with Nadine Lucas: if you have very different types of text, it would probably be a good idea to deal with them differently. I'm not really sure I understand the question about creating the model "down up" vs "top down". Unless I've totally misunderstood, we're discussing an automatic classifier based on some sort of machine learning algorithm? If that is the case, then performance will almost always be better with a bigger data set (compared to e.g. switching algorithms), so starting with a small data set and scaling up might not be the most effective way of going about things.

In training the model (on the annotated test/training set), it would probably be good to have as much information available to the algorithm as possible. And of course, as Nadine Lucas points out: there's plenty of structure in text (linguists tend to wince when they hear text referred to as "unstructured"): morphological, syntactic, semantic, and pragmatic. I would assume such structure would be useful to the model (I think there are NLP tools around for Twitter data, but I don't have the link available right now).

Nadine Lucas, what you say makes a lot of sense. In fact, we've been able to model two types of text - deceptive and truthful - based on their discourse structure features:

Rubin, V. L. and Vashchilko, T. (2012) Identification of Truth and Deception in Text: Application of Vector Space Model to Rhetorical Structure Theory. EACL 2013: Computational Approached to Deception Detection Workshop, Avignon, France.

http://dl.acm.org/citation.cfm?id=2388631

Gard Jenset, we have also looked at deceptive cues at various levels of linguistic analysis that you mentioned:

Rubin V.L. and Conroy N. (2012) Discerning Truth from Deception: Human Judgments and Automation Efforts. First Monday, 17 (3)

http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/3933/3170

Thanks again for your clarifications and suggestions.

Very nice subject. We have designed special algorithms for cross-verification of data and improve data quality, which in our opinion (and business) is crucial.

Lucian, very interesting. What kind of data are you able to verify? What is your approach roughly? Have you written anything about your algorithm that you'd be able to share? I'd appreciate a citation or a link to an "About" page. Thank for you response.

Victoria, we do work with data regarding business entities, most is numeric, but also text. Here are some links

https://www.researchgate.net/publication/228399931_Data-Mining_Techniques_for_Supporting_Merging_Decisions?ev=prf_pub

http://www.google.ro/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fwww.journal.univagora.ro%2Fdownload%2Fpdf%2F637.pdf&ei=yyvmUpDzMbCEyAOP5IDACA&usg=AFQjCNGdhNnp_y7WzL1ib2Jgtsnetjfivg&sig2=vM2qZPiOYfRchK97vculhg&bvm=bv.59930103,d.bGQ&cad=rja

http://ideas.repec.org/a/blg/reveco/vsupplementy2012i5p94-101.html

Regarding deception, we think that interesting approaches can be derived for discovering deception in business transactions as well ;)

Article Data-Mining Techniques for Supporting Merging Decisions

If you consider outlier what doesn't fit in the model (referring to one of the previous comments) you could validate your model even if it's not valid. It all depends on the goals and on the source of the research.

The challenge in Big Data is exactly that your data is not structured and full of "errors", so finding a model which can process this kind of data is the thing of main interest, in my opinion, much more than the opposite (finding data that can be treated by standard models).

Finding a model for messy data requires lot of time compared with standard modelisable data.

While you work with really BIG data, quality is NOT an issue. For example, while taking all street networks of a country, we found that there are far more small street blocks than large ones. This finding is not affected by errors.

Jiang B. and Liu X. (2012), Scaling of geographic space from the perspective of city and field blocks and using volunteered geographic information, International Journal of Geographical Information Science, 26(2), 215-229. Reprinted in Akerkar R. (2013, editor), Big Data Computing, Taylor & Francis: London.

In addition, natural cities derived from social media data are also not affected by errors.

Jiang B. and Miao Y. (2014, accepted), The evolution of natural cities from the perspective of location-based social media, The Professional Geographer, xx(xx), xx-xx, DOI: 10.1080/00330124.2014.968886,  Preprint: http://arxiv.org/abs/1401.6756

Jiang B. (2015a), Head/tail breaks for visualization of city structure and dynamics, Cities, 43, 69-77.