I need to finding duplicate items for discover the Association Rules in the Bigdata. Common algorithms for solving the Association Rules are Apriori, FP-Growth and others...but I don't use these algorithms. I want to improve the finding time of duplicate items, guide me please
One alternative is always to have a new field with a hash of the concatenated fields; when choosing the right hash function, the false positives will be minimum even in big data (and, still, you can check equal hashes field by field to make sure that you have no false positives). Depending on your data, you can also use a probabilistic data structure such as Bloom Field or Locality Sensitive Hashing, in some (rare) cases removing similar entries can improve performance.
There are specific function in R software for identifying duplicated data...
namely unique() or duplicated()
Is this what you are looking for?
the easiest method is two for loops and if the iteam has been found delete it. the complixity of this algorithm is O(nlogn). if you do not delete duplicated iteams it will be O(n^2). This algorithm may take a big time in normal computation. try to do it in parallel programming (threading or GPUs)
You can add a new field to the dataset by concatenating several string fields with a separator, e.g. "~". Than group the dataset by the new field and count cases.
One alternative is always to have a new field with a hash of the concatenated fields; when choosing the right hash function, the false positives will be minimum even in big data (and, still, you can check equal hashes field by field to make sure that you have no false positives). Depending on your data, you can also use a probabilistic data structure such as Bloom Field or Locality Sensitive Hashing, in some (rare) cases removing similar entries can improve performance.
It's a very broad question with no clue. First you need to find the features (columns in your data) that make them the same or similar. Based on your data, you can develop a model to select/calculate an item score to uniquely represent these features. Then, find the score treshold (acceptible score variation) which will minimize the error. So this may save you to compare each row with others.
How big it will be. Can you hash it and compare the hash value? That should be able to process a significant amount of big data.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Algorithms