Dear Tulika,

✪ such scripts are normally custom codes. Thus, even if there is, which most likely is, you might need to perform time-consuming re-checks, re-tests, re-validations, just for dataset preparation.

✪ It may be rational if you use an already prepared dataset to initially design, test and validate your machine learning algorithms or systems, rather than you spend your time to convert and clean dataset(s). Then, if your algorithms worked, you can go back and prepare your own datasets, as an addition, as well, if necessary.

Best,

J.

████████████████████████████████████████████

https://www.wireshark.org/

http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

https://github.com/jadianes/kdd-cup-99-spark

https://github.com/fichtner/flows_to_weka

████████████████████████████████████████████

Thank you J. Rafiee.

Using tshark in Linux, the Command line code used to extract the tcp conversation is

tshark -r -q -z conv,tcp >

The above command takes the pcap or dump file and looks for converstion list and filters tcp from it and writes to an output file in txt format, in this case, in the order of flow sizes.

Recently, we have used Benfords law to extract features and detected anomalies in network flows using machine learning. The article can be found at https://arxiv.org/pdf/1609.04214.pdf . The code is implemented in Matlab and is also available to download from the links in the paper.

A simple shell code for chunk processing a very large tcp-dump is attached as extract_tcpdump.zip.

Wish you all the best.

Thanku so much Santosh Tirunagari

Hi Tulika,

Where you able to create KDD dataset from tcpdump.

As I am also doing a similar project, and I am a bit confused.

@Akash Antony.. I'm sorry. I was unable to do this conversion. I'm still trying.

Hi Tulika,

Thanks for the information. So are you trying to replicate the dataset or are planning to find out new features which would be better than the existing ones?

@Akshay Antony.. planning to find out new features.

We had similar requirements in our research and developed our own tools for this purpose.

If you know the attributes you want to extract, you can use pcap-processor(https://github.com/slgobinath/pcap-processor) to extract, pre-process and to write them to csv file or anywhere else.

My colleague @Nadun Rajasinghe has developed a framework to create KDD like datasets from pcap file. (https://github.com/nrajasin/Network-intrusion-dataset-creator) which may be useful for your research.

Thanks a lot Sir @Gobinath Loganathan

Thanks a lot! Gobinath Loganathan

Tulika Agrawal & Akash Antony

Its my pleasure. If you have any questions regarding pcap-processor, feel free to ask me.

Hi Gobinath Loganathan . I tried running the (https://github.com/nrajasin/Network-intrusion-dataset-creator). But it is generating a lot of errors while installing requirements.txt.

Could you help me with it!

Hi Akash Antony,

Please open GitHub issues in the project itself. Nadun may help you to solve the problems there. I can help you with the pcap-processor. To generate CSV files using the pcap-processor, you can use the following command:

python3 -m pcap_processor --sink csv samples/cicids_2017.pcap

Now I also have added sample CSV files generated from Darpa 99 and CICIDS 2017 datasets.

Hi Gobinath Loganathan while running the pip3 install -r requirements.txt I am curently getting an envirenment error.

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/dist-packages/Logbook-1.4.1.dist-info'

Consider using the `--user` option or check the permissions.

Could you please tell me how it is to be resolved. I am not able to resolve it!

Hi,

Try the same command with sudo like this:

sudo pip3 install -r requirements.txt

It is giving the following error while trying that

Traceback (most recent call last):

File "/usr/bin/pip3", line 9, in

from pip import main

ImportError: cannot import name 'main'

For pcap files you can use Tshark, for example the following line command will read INPUT.PCAP file and retrieve all fields with -e argument with the filter rule in -Y argument using separator "," (CSV) output to OUTPUT.CSV file:

$ tshark -r INPUT.PCAP -T fields -e frame.protocols -e frame.len -e frame.time_delta -e udp.dstport -e udp.srcport -Y "ip.dst == 192.168.1.1" -E separator="," > OUTPUT.CSV

More info: https://www.wireshark.org/docs/man-pages/tshark.html