Dear all,
Do you have idea to resolve zero and negative value issue of a large dataset of PM2.5 and PM10?
I started to explore a big dataset and found many negative and zero values that need to be corrected.
Similar incidents are also observed in gas data (NO, NO2, NOx, CO, CO2, SO2). What I understand that negative values are recorded due to instrumental malfunction or concentration is very low (near detection limit).
I hope I will receive many useful suggestions.
What is/are the instruments you use?
The way you write it down now is much too vague
A general answer could be that you do not calibrate the instruments
My first upload got apparently lost
A very first question to you before any answer can b given is what are your instruments
do you calibrate these regularly; I am leaning towards the idea that you do not because zero and negative results mostly occur when calibration is absent for a long time
Additionally,
When the data were calibrated, then eliminate the anomalies you cited. First the negative data. If you know the detection limit of the methods used to generate your dataset, you can replace the zero data by the detection limit, indicating that a value is equal to the detection limit or lower.
Cheers,
Frank
PS.: Harry pointed out that uncalibrated values are pretty much worthless.
I was not in the field during the campaign. So I talked to those persons did measurement. They did calibration regularly during the campaign. The main problem is the PM2.5 and 10 as I found many negative values. Dusttrak was used during campaign to measure PM1,2.5, and 10.
Do you think Dasttrak produce lots of negative values when the ambient concentration is too low? I found a protocol for data management- http://www.mfe.govt.nz/publications/air/good-practice-guide-air-quality-2009/html/page10.html
Thanks Harry and Frank for your suggestion. Once I will get more update on instrument calibration procedures they followed, I will post again.
Warm regards,
Mahmud
I think that the best thing to do, if you cannot, assume those values as non-valid ones, is to consider all negative or zero values as equal to the detection limit of the measuring method.
The dusttrak is not accepted as an instrument for compliance measurements for PM
Standard are the TEOM or beta-gauge.
The dusttrak uses light-scattering and this by definition cannot give negative values. However, the instrument has a dark current and other blank signal that have to be subtracted. Is this atomatically doen and how is this blank measured; by a filter at the inlet and how often?
Cause of negative PM, gas values: inappropriate low frequency/low quality of zeroing, may also frequently appear when the contractions are near the instrumental detection limits. 0 cocn. or less does not exist in principle. First look at the basic statistics of the dataset. Sometimes is acceptable to work with the dataset where 10% data must be manipulated sometimes it is not acceptable at all. Anyway, this data manipulation must be mentioned when presenting, evaluating them.
solution: 1. discard the dataset. This depends on set-up of probability you are working on, and data distribution.
2. replace 0 or
I like to repeat and emphasise that when this occurs over a long period of time there is a systematic error in the calibration and the values cannot be trusted whatever statistics you try to append on the data
I definitely agree with Harry. Systematic error excludes dataset utilization, nevertheless stochastic error allows the analysis procedure as described.
Thanks Dr Jan and Harry for wonderful suggestions, with great explanation from different angle.
Calibration was conducted weekly basis. So I hope it will not be so hard for me to identify obvious outliers.
Also data was recorded every 30 seconds so their is ample scope to exclude limited outliers.
The Value of oxides of nitrogen depends on temeparutre, availability of oxygen in side engine cylinder, fuel, stroke etc. Hence calibration ia required with reference gas for instrumentation and especially repeatablity tests are to be conducted in order to ensure working of gas analyzers.
- Badges
- Science topic
- Similar topics
- Engineering
- Petroleum Engineering
- Gas