To perform data quality assessment in the pre-processing data phase (Big Data Context), should data profiling being performed before data sampling (on the whole data set), or is it ok to have profiled on a subset of the data?
If we consider the second approach, how sampling is done without having information about the data (even some level of profiling)?
It depends. If your subset has enough categories that are in the whole dataset, then the data profiling can be very efficient. But, for instance, when you have 10 categories in a certain feature, and only 1 category is in your subset, then you won't get any new statistics.
Try to assess how much data for the subset is required to perform profiling.
Even with a small subset of data, you can observe what is the quality of your data.
Taras Ustyianovych Based on that, we can say that we need a primary data profiling to get the information required for the sampling process, then data sampling will be performed. After data sampling is done, the complete data profiling operation is executed over the data samples.
Hadi Fadlallah , yes. That should decrease computational expenses, help to perform an investigation of a subset instead of the whole set. That is similar to a data science process where a small dataset is analyzed afterward the methods are applied on big data set.
We cannot say exactly, refining data sampling and data quality assessment would be done based on need and raw data and other requirements.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data