I am conducting research using survey-based datasets (specifically NFHS-4 and 5) to estimate breastfeeding patterns and the time interval between marriage and the first birth. I am employing regression-based survival models, specifically semi-parametric and proportional hazards models. The dataset has a substantial sample size (N) of approximately 150,000, and due to computational limitations, the models cannot be applied to the entire dataset.
I would like to inquire about the possibility of utilizing bootstrap methods (or any suggested ones) by treating N as the original population. This would involve drawing samples of reasonable sizes (e.g., 1,000 or 5,000) from the population and subsequently estimating the desired parameters using proposed models along with their corresponding standard errors.
Thank you Dauji Saha
for your suggestions and views.Currently, I am exploring different methods that are most suitable and will try your suggested approach as well.
I think different methods are data-specific also, so better to explore all possible methods to find the optimal one.
Yes, resampling methods involve repeatedly drawing samples from the original dataset, estimating parameters on each sample, and then aggregating the results to obtain parameter estimates.
One commonly used resampling technique is bootstrapping. Bootstrapping involves drawing random samples with replacement from the original dataset to create multiple bootstrap samples. Parameter estimates are then computed on each bootstrap sample, and the distribution of these estimates is used to infer the population parameter's uncertainty. Bootstrapping is particularly useful when there is no simple analytical solution for estimating the parameter or when the distributional assumptions required for traditional statistical methods are violated.
Another resampling technique is cross-validation. Cross-validation is commonly used for model evaluation and selection but can also be used for parameter estimation. It involves splitting the dataset into multiple subsets (e.g., training and validation sets) and iteratively estimating the parameters on different subsets. By averaging the parameter estimates across multiple iterations, more robust parameter estimates can be obtained.
It's computationally intensive, especially for large datasets. In such cases, there are also alternative methods for parameter estimation in large sample real datasets, such as:
1. Subsampling: Instead of resampling the entire dataset, subsampling involves randomly selecting a smaller subset of the data to estimate parameters. This can reduce computational burden while still providing reasonable estimates.
2. Distributed Computing: Large-scale parameter estimation can be performed using distributed computing frameworks such as Apache Hadoop or Apache Spark. These frameworks allow for parallel processing and distributed computations, enabling efficient analysis of large datasets.
3. Summary Statistics: Instead of estimating parameters on the entire dataset, summary statistics can be computed to capture relevant information. This can reduce the dimensionality of the problem and make parameter estimation more manageable.
4. Approximation Techniques: In some cases, approximation techniques, such as variational inference or stochastic gradient methods, can be used to estimate parameters in large datasets. These methods provide computationally efficient approximations to the true parameter estimates.
Good luck
partial credit ai tool
Here is the reviewer's comment we obtained when using the resampling approach: The data analysis presented in the paper is very strange indeed. The authors use the NFHS data consisting of N (say 150,000) cases, but confine their analysis to 2 subsamples each of size 1000, where the proportion of right-censored cases, precisely recalled cases and cases of non-recall are controlled [three categories considered in my study]. This reduces the exercise into something similar to a simulation study, without the benefit of knowledge of the generating model.
For such a large sample, any parametric model with a handful of parameters is likely to produce an inadequate fit. The authors could have chosen a smaller data set, or restricted their analysis to a filtered segment of the NFHS data set, for the purpose of illustrating their method. Choosing a synthetic subsample from large data hardly makes their case convincing.
Yes, employing bootstrap methods is a reasonable approach when dealing with large datasets and computational limitations.
- Badges
- Science topic
- Similar topics
- Mathematics