I am working on a project that is likely to predict and analysis humungous rasters and datasets. The largest I have worked with are 15 times smaller. So, it would help if anybody could suggests any studies or publications who done analysis with such large raster. It would be great, if you could share your experience too and how you may have overcome the size of the rasters in your analysis.
Dear Patricia Ryser-Welch
While 8GB of RAM is adequate, 16GB or more is preferable for most deep learning projects. A minimum of 7th generation (Intel Core i7 processor) CPU is suggested. Getting an Intel Core i5 with Turbo Boosts, on the other hand, may suffice.
Kind Regards
Qamar Ul Islam
Can you give more information about the concrete problem you are trying to solve? The first thing I think about when we are talking BIG images is a tiling mechanism that would split the image into more manageable chunks of a size that is digestible on the hardware you have (might be less than 1Gb tile).
Depending on the problem you are trying to solve, like classification, segmentation, etc. it might do the trick. Another advantage is that you could distribute the processing of each tile on a different computers to speedup your experiments.
On the implementation side, even reading the original image in order to split it is a programming challenge where you probably need some "streaming" custom reading of the file to read a few Gb and write it as a tile and so forth. These large "stripes" of the original image could then be splitted into smaller more square tiles if you need to. Depending on your original image format, that might be relatively easy or very difficult (i.e. if encoded by complex compression scheme like JPEG)
Also, before coding all this by yourself, I would look for available tools from the remote sensing and satellite imagery domain. These guys have encountered this problem before and probably already have tools to deal with it. I'm sure not only Google Earth developers has solve this...
Good luck!
It really depends on the type of model and the input size of your model. For example, I worked with a dataset of ~100GB of satellite images and it was possible for me to train a complex CNN using my personal workstation (64GB, NVIDIA Quadro) because the model required small input patches of 64x64x4 pixels. However, for another project where I needed to process videos of 128x128x100, I needed to process it in an HPC because it was not possible to train it even with small mini-batches. This dataset was smaller but the model was more complex. The lesson is that the size of the dataset does not determine your hardware needs.
Hi Patricia Ryser-Welch ,
I currently have a test bed in Java which is dealing with terabytes of data, as required. Maybe we could have a chat?
However the basic principle is that you create a "frame" and read the size of the data which fits the frame into memory and then walk the rest of the dataset.
My particular solution is able to use disk-based buffering if the local memory is not large enough. However essentially what I'm doing with the source data is walking a result set and reading in the source data frame by frame.
It would be helpful to have a little more detail about what you're trying to do though? For example how much data you need to hold in memory at any given time. I'm guessing that if you're training as CNN it's actually not that much… The size of the total data storage for the current state of the CNN?
Best Wishes
Ian
- Badges
- Science topic