Any suggestions on how to parse such large pcap.gz files of telescope data from caida.
If I use tcpdump or tshark, it is taking too long to parse them.
Also, I have a big list of filter prefixes, tcpdump throws an error of too long filter file. How to take care of that?
I'm not sure what you mean the data is live if it is in a pcap file.
If it is on the wire, you can still write code to parse it on the fly off the wire or you can use the same code to read the packets out of a capture file which is good for testing.
Are you trying to reassemble data arriving in multiple packets and then write the reassembly to files?
It's relatively easy to do if you know the start and end markers.
I mean, the each file get generated hourly, and gets deleted within a few hours. Even reading with tools like tcpdump, tshark, can process only about a few minutes of data before it gets deleted.
Thanks! Another issue is that these files are generated on an ISP server, so can't play much on those machines.
- Badges
- Science topic