Dear colleagues,
I'm using Matlab for writing the output from some computations to quite large NetCDF-Files. I have encountered a problem when multiple "workers" or processes try to write simultaneously to the same NetCDF-File. This, however, does only happen when using compressed and chunked NetCDF-Files; i.e. if the file is created without any compression or chunking, the problem does not occur.
However, writing to an unchunked file takes much more time. In my case, writing to unchunked files takes about 30 seconds compared to only about 0.1 seconds when the files are reasonably chunked.
Therefore, I would very much prefer to write to chunked NetCDF-files as this speeds up the computations significantly. Do you have an idea how a NetCDF-File must be created in order to write to the same chunked file using multiple processes or workers?
Best regards and many thanks in advance!
Even if I've posed this question more than one year ago, I just wanted to give a quick update on this issue as this might be of interest for other Matlab users as well.
As far as I know, writing to a single NetCDF-file from multiple workers is always quite problematic. Therefore, I have tested two "workarounds":
- Create a separate file for each worker and merge these files afterwards or
- (from Matlab 2017b) use the DataQueue from Matlabs Parallel Computing Toolbox (https://de.mathworks.com/help/parallel-computing/parallel.pool.dataqueue.html)
After several tests, I found the second solution to be much more straightforward (even if it might take slightly longer). If you have any questions about the implementation, please let me know.
Cheers!
- Badges
- Science topic