Classification is not a fully automated process from start to finish.

In order for a search engine to classify acquired data from crawled pages it needs to have a pre-provided classification lookup table, or relevance dictionary for keywords and phrases (if data is textual), or other values adequate for indexed data.

This initial dictionary is something you have to manually define, or generate by using some sort of statistical tool to: First, analyse existing data (samples): Second, Extract keywords or parts of data by highlighting it (you may choose to use regular expressions for highlighting): and Third: Manually classify extracted data into groups by assigning relevance factors (what is more important and what is less important), initial search keywords for that particular relevance factor for the particular data set and so on. I believe you understand what I mean to say and what is to be done.

After this is done, you can build an automated crawler that will collect meta data or/and extract content data by use of meta-data dependant regular expression choice (defined in the dictionary, for better and more relevant extraction process) and then analyse it by pairing the extracted data by means of "similarity" to that defined in the dictionary and assign relevance factors and then store with proper indexing in the database.

Make sure you save the highlighter and extracter temporary variable values, such as positions of extracted data within page, because you can use this in the presentational layer to provide nice highlighting of the most relevant, less relevant and irrelevant parts of the page content (coloured differently or in a different manner).

When crawling is finished and all pages have been stored, valued with relevance to a set of keywords (all keywords found in the current page's meta data container tags) and indexed, you can preform searches.

When someone searches for certain keywords or values (if numeric data has been indexed), you can use normalization to better the search input.

When I say normalization, I mean to discard of unnecessary parts of the input string and in case your language has cases, strip down the keyword to nominative case (this should be done while indexing and storing into the database as well).

Then tokenize the input to allow better union vs intersection choice of result grouping and to assign a larger factor to the relevance index for results provided for the keyword placed more to the front of the search input (prioritize it).

I hope this intro gives you some ideas on where to start and what to watch out for.

Regards,

Milan

Thank you for your explanation, Sir.

What I was asking is how to classify the result of wrapping in meta-search engine (xml file), I don't use we crawling, since I wish to build a meta-search engine..

Do I use a text classifier instead of web classifier?

Thanks again for your helpful response!

For that you can use one of existing document classification SDKs such as uClassify. It is a good document classifier project. It is provided as a SDK for a number of languages.

http://www.uclassify.com/ApiDocumentation.aspx

Does uClassify support PHP, Sir?

Then, what do I do to build a wrapper for the SERP? Can I use PHP? But I still don't know how it works..

Thanks again.

On git-hub, there are two projects, or rather, implementations of uClassify in PHP. They are published as libraries, and are Open Source.

One is https://github.com/ashwanthkumar/uClassify-PHP5 and the other https://github.com/markdthompson/uClassify-Utility

The first provides the use of text classification based on contextual "understanding" and the second has a short demo that demonstrates weighting results.

As for the result page, you can use foreach loops to iterate through the result set and then display results in any form you wish to.

You could do this in PHP directly by echoing data into HTML or XML, or you can just serialize the result array data into JSON and parse it in another application (Desktop app, Android app, etc.) it doesn't have to be a PHP app all the way down to the end.

Hi, I have done some work in a related area on searching for open educational resources (OER). I have used text mining and clustering instead of classification. It might be of some interest to you.

The framework : https://www.researchgate.net/publication/236170282_Conceptual_Framework_for_Parametrically_Measuring_the_Desirability_of_Open_Educational_Resources_using_D-Index?ev=prf_pub

The initial prototype search engine: https://www.researchgate.net/publication/236170320_OERScout_Autonomous_Clustering_of_Open_Educational_Resources_using_Keyword-Document_Matrix?ev=prf_pub

Article Conceptual Framework for Parametrically Measuring the Desira...

Conference Paper OERScout: Autonomous Clustering of Open Educational Resource...

Thank you very much for the answers! :)

I'm working on it..