Hello
I have the following situation: I have a paper X about topic Y. For paper X I did a forward search with Web of Science (checking all new papers which cite paper X). Then I have downloaded all articles I have identified via forward search (approx. 1'000 Papers). Now I would like to sort these papers according to the frequency of specific keywords used.
For example: I have found paper Z via forward search (so paper Z cites paper X which is about topic Y). Now I want to check if paper Z is also concerned about topic Y or if it just refers to it in passing. For that I search for specific keywords which correspond to topic Y. According to the frequency of the specific keywords mentioned in paper X, I want to classify it in the category "relevant" or "not relevant". Now, how can I determine the threshold for the keywords? That is, if paper X only uses the specific keyword once it is most probably not relevant to topic Y. But if it mentions the specific keyword 20 times it is probably relevant for topic Y.
Is there a recognized methodology to determine or approximate a threshold for the keyword frequency which allows to distinguish if a paper is relevant to topic Y or not?
With this approach I hope to reduce the 1'000 papers to those which are about topic Y.
Related to you query you may follow : https://www.quora.com/What-is-the-threshold-word-frequency-that-makes-a-word-crucial-to-learn
https://pdfs.semanticscholar.org/7844/e5f18a3dc8520514e5dc532b968d2eb74359.pdf
https://www.wordstream.com/keyword-frequency
It is possible that ASK-analysis and the Eidos system will be suitable for solving Your problem.
As a method of the research we used Automated system-cognitive analysis (ASC-analysis), which is a new innovative method of artificial intelligence: it also has its own software tool – an intelligent system called "Eidos" (open source software).
The Eidos-X++ system differs from other artificial intelligence systems in the following parameters:
- was developed in a universal setting, independent of the subject area. Therefore, it is universal and can be applied in many subject areas (http://lc.kubagro.ru/aidos/index.htm);
- is in full open free access (http://lc.kubagro.ru/aidos/_Aidos-X.htm), and with the relevant source texts (http://lc.kubagro.ru/__AIDOS-X.txt);
- is one of the first domestic systems of artificial intelligence of the personal level, i.e. it does not take special training in the field of technologies of artificial intelligence from the user (there is an act of introduction of system "Eidos" of 1987) (http://lc.kubagro.ru/aidos/aidos02/PR-4.htm);
- provides stable identification in a comparable form of strengh and direction of cause-effect relationships in incomplete noisy interdependent (nonlinear) data of very large dimension of numerical and non-numerical nature, measured in different types of scales (nominal, ordinal and numerical) and in different units of measurement (i.e. does not impose strict requirements to the data that can not be performed, and processes the data that is);
- contains a large number of local (supplied with the installation) and cloud educational and scientific applications (currently 31 and 200, respectively) (http://lc.kubagro.ru/aidos/Presentation_Aidos-online.pdf);
- provides multilingual interface support in 44 languages. Language databases are included in the installation and can be replenished automatically;
- supports on-line environment of knowledge accumulation and is widely used all over the world (http://aidos.byethost5.com/map5.php);
- the most time-consuming computationally, the operations of the synthesis models and implements recognition by using graphic processing unit (GPU) that some tasks can only support up to the solution of these tasks is several thousand times that really provides intelligent processing of big data, big information and big knowledge;
- provides transformation of the initial empirical data into information, and its knowledge and solution using this knowledge of classification problems, decision support and research of the subject area by studying its system-cognitive model, generating a very large number of tabular and graphical output forms (development of cognitive graphics), many of which have no analogues in other systems (examples of forms can be found in: http://lc.kubagro.ru/aidos/aidos18_LLS/aidos18_LLS.pdf);
- well imitates the human style of thinking: gives the results of the analysis, understandable to experts on the basis of their experience, intuition and professional competence.
See also my work on ASC analysis of texts (priority for newer ones).
References to prof. E. V. Lutsenko's work on text analysis
Ask-text analysis allows you to:
- generate generalized linguistic images of classes (semantic cores) based on fragments or examples of related texts in any language;'
- quantitatively compare the linguistic image of a particular person or description of an object or process with the generalized linguistic images of groups (classes);
- compare generalized linguistic images of classes with each other and create their clusters and constructs;
- explore the simulated subject area by examining its linguistic system-cognitive model;
- conduct intellectual attribution of texts, i.e. determine the likely authorship of anonymous and pseudonymous texts, Dating, genre, and semantic orientation of the text content;
- all this can be done for any natural or artificial language or coding system.
Lutsenko E. V. Synthesis of semantic cores of scientific specialties of the Russian Academy of Sciences and automatic classification of articles by scientific specialties using ask-analysis and the intellectual system "Eidos" (on the example Of the scientific journal of Kubgau and its scientific specialties: mechanization, agronomy and veterinary medicine) / E. V. Lutsenko, N. V. Andrafanova, N. V. Potapova / / Polythematic network electronic scientific journal of the Kuban state agrarian University (Scientific journal of Kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2019. – No. 01(145). Pp. 31-102. – IDA [article ID]: 1451901033. - Access mode: http://ej.kubagro.ru/2019/01/pdf/33.pdf, 4.5 C. p. l.
Lutsenko E. V. Formation of the semantic core of veterinary medicine by Automated system-cognitive analysis of passports of scientific specialties of the HAC of the Russian Federation and automatic classification of texts in the areas of science / E. V. Lutsenko / / Polytematic network electronic scientific journal of the Kuban state agrarian University (scientific journal of kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2018. – No. 10(144). P. 44 – 102. – IDA [article ID]: 1441810033. - Access mode: http://ej.kubagro.ru/2018/10/pdf/33.pdf, 3,688 C. p. l.
Lutsenko E. V. Intellectual binding of incorrect references to literary sources in bibliographic databases using ask analysis and the "Eidos" system (on the example of the Russian scientific citation index-RSCI) / E. V. Lutsenko, V. A. Glukhov / / Polythematic network electronic scientific journal of the Kuban state agrarian University (scientific journal of kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2017. - No. 01 (125). C. 1 – 65. – IDA [article ID]: 1251701001. - Access mode: http://ej.kubagro.ru/2017/01/pdf/01.pdf, 4,062 C. p. l.
Lutsenko E. V. Application of ask-analysis and intellectual system "Eidos" for solving the problem of identification of literary sources and authors by standard, non-standard and incorrect bibliographic descriptions in General form / E. V. Lutsenko / / Polytematic network electronic scientific journal of the Kuban state agrarian University (scientific journal of kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2014. - No. 09 (103). S. 498 – 544. – IDA [article ID]: 1031409032. - Access mode: http://ej.kubagro.ru/2014/09/pdf/32.pdf, 2,938 C. p. l.
Lutsenko E. V. ask - analysis of the problems of articles Of the scientific journal Kubgau in dynamics / E. V. Lutsenko, V. I. Loiko / / Politematic network electronic scientific journal of the Kuban state agrarian University (Scientific journal of Kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2014. - No. 06 (100). P. 109 – 145. – IDA [article ID]: 1001406007. - Access mode: http://ej.kubagro.ru/2014/06/pdf/07.pdf, 2,312 C. p. l.
Lutsenko E. V. Attribution of anonymous and pseudonymous texts in system-cognitive analysis / E. V. Lutsenko / / Polytematic network electronic scientific journal of the Kuban state agrarian University (Scientific journal of kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2004. – No. 03 (005). Pp. 44 – 64. – IDA [article ID]: 0050403003. - Access mode: http://ej.kubagro.ru/2004/03/pdf/03.pdf, 1,312 C. p. l.
Lutsenko E. V. Attribution of texts as a generalized problem of identification and forecasting / E. V. Lutsenko / / Polytematic network electronic scientific journal of the Kuban state agrarian University (Scientific journal of kubgau) [Electronic resource]. - Krasnodar: Kubgau, 2003. – No. 02 (002). P. 146 – 164. – IDA [article ID]: 0020302013. - Access mode: http://ej.kubagro.ru/2003/02/pdf/13.pdf, 1,188 C. p. l.
Lutsenko E. V., Grushevsky S. P., Grushevskaya T. M. Intellectual attribution of literary texts (text Dating, authorship and genre definition on the example Of Russian literature of the XIX and XX centuries) / ResearchGate, 2020, DOI: 10.13140/RG.2.2.27935.92324,
Dear David Redaschi ,
Look the link, maybe useful.
https://www.semrush.com/lp/keyword-magic-see/en/?kw=%2Bof%20%2Bkeyword%20%2Bresearch&cmp=WW_SRCH_Keyword_Magic_Tool_set3_broad_EN&label=keyword_magic_tool&Network=g&Device=c&utm_content=408190356844&gclid=Cj0KCQjw0pfzBRCOARIsANi0g0tY3jSoJPHFUj2Qn1znTmFhmBa4kbS_s2eRNaCjfdaNvSGUX28s8_AaAtQ9EALw_wcB
Regards,
Shafagat
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining