Data quality testing is the practice of making assertions about your data, and then testing whether these assertions are valid. This concept can be used to test both the quality of your raw source data and to validate that the code in your data transformations is working as intended. According to Gartner, bad data costs organizations on average an estimated $12.9 million per year.
Dear doctor
I quoted the following from web hoping being satisfying
"Traditionally, data engineers write data quality rules using SQL. This manual method works well when there are dozens, or even hundreds of tables, but not when there are ten thousand or more. In a modern, data-driven organization, data engineers can never keep up with demand for data quality scripts.
New data quality automation (DQA) tools replace manual methods with ML models. You can view this product segment as a subset of data observability, which addresses both data quality and data pipeline performance. There are three different approaches to DQA: automated checks, automated rules, and automated monitoring. Each approach has its pros and cons, but collectively they represent the future of data quality management.
The following three vendors embody these approaches, respectively.
- Ataccama employs the automated checks approach, which uses ML to classify incoming data at the row and column level and automatically apply data quality rules written by data engineers.
- First Eigen uses the automated rules approach, which uses ML to generate data quality rules, which consist of standard quality checks (nulls, duplicates, etc.) and complex correlations between data columns and values.
- BigEye uses the automated monitoring approach, which uses ML to detect anomalies in the data rather than apply rules. The tool monitors changes to tables at fixed intervals, triggering alerts if it detects an uncharacteristic shift in the profile of the data."
Dr.Sundus Fadhil Hantoosh
Very nice query? In such era & in every field the data is generated, stored & used for future plans. And the most important point is quality of data, if the quality is poor the results would be poor. Data in health management system should be of high quality for present & future plans but, unfortunately, the quality remains in poor because the data managers care minimum on data quality. So, the data/manager should be well qualified, trained, adept in handling the data & keep watch in continuum ie see that the data/information is coming timely, correct, complete, regularly & provide feedback to who generated data. Continuous monitoring, comparing previous information, cross-check & on the spot evaluation of data where generated. Maintenance of Data quality is
really a huge task & more focus is necessary.