I am looking for data analysis tools that can be added into a new database (legacy system migration, without database from legacy system) which takes structured data (pre-determined format, seen as correct) as an input.
That depends on many things e.g. the nature of data, how data quality is defined and what systems are being used. Systematic data validations could be a good remedy. For the analysis part, you may want to add few data exploratory options for visual data quality checks.
Depending of the systems, you can deploy AI based autonomous anomaly detection methods to ensure quality as well.
Nowadays, most data scientist assess the quality of their data at the moment they check whether the statistical model they are using is well-suited to the problem. There exist multiple goodness of fit tests that can offer you a good understanding about how your data (observed values) are correlated to the expected values under the model. Therefore, you can decide if the data you manage is sufficiently representative for your work task.
However, this approach is not the most efficient when you want to boost the data quality of your data warehouse. In that case, the best strategy would be to define a set of points (protocols) that your input data needs to meet to be taken into further consideration. To carry out this you can apply a wide range of rules and validation methods:
- Size/format checks, consistency checks, file existence/missing values, uniqueness check, limit check, ...
There are many data analytic tools (usually commercial) that claim that they can provide information about the quality of your files (and some also claim that they can subsequently provide files where the errors are 'corrected'.) The methods are called 'profiling' tools. I am not aware of any that work in a minimally effective manner. Difficult errors are determining 'duplicates' in files using quasi-identifying information such as name, address, date-of-birith, etc. Two records may be duplicates (represent the same person or business) even when the quasi-identifying information as representational or typographical error. Any quantitative information from the two records representing the same entity may have slight (or major) differences. If the records having missing values associated with the data that an individual wishes to analyze, then the missing values should be filled-in with a principled method that preserves joint distributions (e.g., Little and Rubin book on missing data, 2002). The 'corrected' data may also need to satisfying edit constraints (such as a child under 16 cannot be married).
htpps://sites.google.com/site/dinaworkshop2015/invited-speakers
- Badges
- Science topic