what level of maturity we have not even scratched surface of it and in NLP is miles to go before you can say that we have neared and big data is a mess and a bliss, mess since no once is clear what is best way and there is nothing this is it, it is continuous research and number of things need to be under this arc and the bliss is supporting technologies are being developed and so....

Hi! It depends on what you would like to do and on which language and on which domain your are working on. If you focus on English that's already an advantage. There are resources, pre-trained models and existing tools out there. Those tools for instance can assist you extracting some pre-defined entities, and for this you can probably get some insights how good those models work - at least on the data they have been tested. I think various information can be extracted quite reliably. On slightly different medical data, this can of course vary. Unfortunately, we are far away from achieve a mature level - whatever we would define as "mature". This is also due to the lack of existing resources here. However, there is much research out there.

Thanks for your replies, the problem relates to cataloguing medical letters and potentially extracting data from them, turning it into a live database.

Roland can I ask why resources or pretrained models you might point to.

Appreciate the interest thus far

Not sure what you mean by "cataloguing medical letters". If you talk about "extracting data from them" and "turning it into a live database", you might be able to use tools such as cTakes or MetaMap to at least extract relevant concepts. As far as I remember, both tools also normalize against UMLS which could help you to put the information into a DB.

However, besides existing tools, which are already established and you can possibly find various information, people might share their models (maybe on github). This could be for instance a particular NER model you can just take and re-use or (if you are lucky) you can re-train it and optimize it to your needs. Regarding pre-trained models there is also a clinical BERT model, you can use.

And regarding resources: what I meant was, that it makes your life easier, if there are resources available (other clinical text datasets). Ideally those would include information/annotations you are interested in. Firstly, you can therefore use this additional data for evaluation or just increase your training data. Second, you have got a better comparison how well a model should perform on your data.

Please, do not overuse Deep Learning. It can't help you so much

Hi Franklin, thanks for your comment, would you care to elaborate? I guess the idea is to see where the boundaries are and look to push at them, right?

Dear Ahad,

I speak to you as Computer Scientist in AI and not as Journalists or whosoever. Deep Learning brought its own contributions, but now we clearly know what it can do and is about. A lot to say, but I just give you some hints:

A Mail Spamer will not classify a mail as suspicious because it understood the sender's intentions or the message in the mail, but rather because it is statiscally closed to some spams.

That's very helpful and I appreciate your time in delineating the issues.

For my purposes I'll probably be looking at its use in this problem as an aid to classify some structured and unstructured medical texts.

Dear Ahad,

What is concretely the tasks? Can you pose the problem as the recognition of statiscally meaningfull pattern?