We have made almost 400 laser experiments where the measurements are done through a photocell, registering voltage variations corresponding to variations in the intensity of the laser beam. When the raw data was plotted, we noticed a "raining" effect, meaning that there was a general line and often from that line came straight down several data points (see for example the Figure attached to this explanation, which includes a base line and the experimental line)
Upon close examination, it was found that the "raining" effect was because Arduino, the laptop, or combination of both, registered the same time stamp for different data. The parsing frequency is 1/100 per second or 100 registers per second. The time stamps do have a resolution of 1/1000th second (for example, 11:09:06.676 -> 332, where the three last digits before the arrow represent tenths, hundredths and milliseconds, but still we get the said "raining" phenomenon).
Here is the question: how to properly calculate statistics with data (time series) repeated in the same time stamp?
Up to now, the descriptive statistics of the data were performed "as is", but later we wondered if the "raining" effect should be removed. The way I came with to remove the rain was to calculate the median of data within the same time stamp (sort of inspired by what it is done in quantum calculations), and then we would have graphics without rain, but then I realized that this entailed that all statistics should be recalculated to account for this modification. So, the choices I am facing, which are the motivation of this question are:
To me, either option 0) or 2) would be consistent in the way of handling data, but I do not discard 1), but the point remains on whether removing the "raining" and substitute it with the median across the subset of points would be a good idea, at least for the graphical representation for the data.
I believe in this context using a noise (or rain effect) filtering tool is acceptable and recommendable. But maybe assigning the median is not the most exact approach, you could try an interpolation approach, such as Savitzky-Golay filtering. And once you filter it, I would do the statistics with it too, not only the visualisation, as it is a correction of the data closer to reality than the original data.
When dealing with time series data where multiple values are recorded for the same timestamp, you need to decide how to handle these repeated values in order to calculate meaningful statistics. Here are a few approaches you can consider:
1. **Aggregate the values**: If you have multiple values for a given timestamp, you can aggregate them using a statistical measure such as the mean, median, sum, or maximum value. This approach is suitable when you want to condense the data into a single value per timestamp. For example, if you have multiple temperature readings for a specific hour, you can calculate the average temperature for that hour.
2. **Use the first or last value**: Another approach is to consider only the first or last value recorded for each timestamp. This is appropriate when the first or last value is more representative or important than the others. For instance, if you have multiple stock prices recorded throughout the trading day, you might choose to use the closing price as the representative value for each day.
3. **Calculate statistics for each timestamp**: Instead of aggregating the values, you can calculate statistics for each timestamp separately. This approach allows you to retain the individual values and analyze their distribution over time. For instance, you can calculate the mean, median, standard deviation, or any other statistical measure for each timestamp independently.
4. **Consider time intervals**: If the repeated values represent different observations within a time interval, you can calculate statistics for each interval. This approach is useful when you want to analyze the data at a higher level of granularity. For example, if you have multiple sales transactions recorded within an hour, you can calculate the total sales amount for each hour.
The choice of which approach to use depends on the specific characteristics of your data and the analysis you want to perform. It's important to consider the underlying assumptions and implications of each method to ensure the resulting statistics accurately represent your data.
- Badges
- Science topic
- Similar topics
- Linguistics
- Composition Studies
- Writing