Hi Ob-

You may check this paper for more info related to ADFA-LD

"Evaluating host-based anomaly detection systems: A preliminary analysis of ADFA-LD"

and URL:

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6743952&isnumber=6743849

Hi Ihab,

thanks for your response. I have read that paper, but it didn't provide the details I needed. I reached out to the author of the dataset, and he was able to help By sending the labels I needed. Thanks.

Hi Obinna,

Can you please share the labels?

My email: [email protected]

Thanks in advance

Ok Mutaz.But note as the label is in a .h file gotten from the system used for generating the ADFA-LD. So, you need to do a little work of extracting it from a ".h" file. Anyways, I'll explain in the email. 

Ok Mutaz. I sent it to your email

hi obinna, is ADFA LD dataset is available in pcap format...?

No @Manmeet.

(obinna) okay, where i can find pcap format if i want to analyze or is there any other latest dataset for Network based IDS testing, i want to test on BRO IDS for anomaly detection.

@ Manmeet, follow the attached link to an answer on Quora with links to recent data sets.

https://www.quora.com/Where-can-I-get-the-latest-dataset-for-a-network-intrusion-detection-system

@Obinna,

That would be awesome I you send me those labels for ADFA-LD. I really need them for my thesis.

Thanks

@ Ehsan,I have attached the file here. I think it is similar to the .h files found in the Ubuntu kernel source code. Anyways that's blah blah blah. Just like I told Mutaz, you will have to do the extraction yourself (you will see syscalls and numbers associated to them in the file attached just do the extraction).

@ Obinna : Can you please send me the details of how to perform the extraction on ADFA-LD dataset. My e-mail ID is [email protected] . Thanks in advance

Ok for any other person requesting for the labels or how to extract them. Firstly, you can still use the unlabeled form of the dataset for training your models or for your algorithms as all you need is basically the class (i.e the attack type). But if you really want to know what each field in the dataset means, then you need to find the .h file that I talked about in this thread. The numbers you see in the dataset are syscalls that were generated or called when Creech (the person who generated the dataset) ran the attacks.  So if you really need a labeled version of this dataset, you need to look at the .h file for the integer number for the dataset and see what name is given to such syscall, then you can say, count the number of system calls with this name e.g if [110 111 110 665 888 999 555 110 110 110 110] is a line in one of the files in the ADFA-LD, and you found out that "110" means "syscall X" (i.e you found out from the .h file), you can say decide to run a little code to extract something like "syscall X" = 6 from the above example, which means that there was 6 "syscall X"s in that line. Based on this, you can move on to whatever. You can see that from my explanation, you might not even need to know the syscall names before being able to use the dataset. 

@Basant  and Prathyusha take note of my last comment. Thanks.

Thanks Obinna for the syscall list + instructions for extracting the labels!

You are welcome kp Pint. I will be writing a research paper soon that will use that dataset. Hopefully, it will shed more light on how you can painlessly extract the dataset. 

Sounds great Obinna! I will keep an eye on your updates here on researchgate. Best of luck!

Where can I get UNM dataset for intrusion detection

After a quick google search I came across this site. Didn't open up the trace files but it looks like this is what you are looking for:  https://www.cs.unm.edu/~immsec/systemcalls.htm

Hi @Balasaraswathi, you know you ought to start your new question about unm dataset on a new thread. This will enable people to be able to find your question easily when they need help, and it will enable you receive useful answers from researchers across Researchgate. 

Could you also send me the details on how to extract those labels? my email is: [email protected]

Thanks in advance

Hi Yellis, the details is in the comments threads of this post. Scroll up a bit and you will see my long comment and attachment on this thread. Thanks.

Hey, just a quick update. An updated version of these datasets has been made available.

Yeah that's true Kp Pint. I just got an email from Professor Jiankun - the creators of the dataset.

When I looked at the updates, the ADFA-LD dataset still remains the same i.e no actual update can be seen in it. I believe the creators of the dataset made the update to point out or create an awareness for 2 or 3 more new datasets that they just released. 

Yep, sorry. I meant that new datasets were added to the ADFA "family". 

@Obinna lgbe. 

Thank you so much for your useful comments. I am going to use the dataset. I would appreciate if you let me know the answer of my questions.

1. The dataset consists of three group of data, training, validation and attack folders. Training and validation folders contain normal process. Am i right?

2. in this dataset (ADFA-LD) there is a system call number 377 which i could not find any corresponding syscall in the current linux system call tables. do you have any explanation for this issue?

thanks

@Sirvan, yes you are right. The Training and validation contains normal processes. Yeah I could not also find some syscalls in the syscall.h file either. The syscall.h file was from a similar system used in generating the dataset, but not the exact same one. that means that not all the applications and processes running in the system used for the dataset generation is running on the similar system that owns the attached syscall.h. I reached out to them in this regard much earlier when I was considering this dataset for a research, and they acknowledged that this was an unlabeled dataset and hence, the syscalls meanings are not also written. Then one of the generators of the dataset -Greeche sent me that syscall which still wasn't from the exact system, but a close approximate. But just like he told me, and like I am telling you and everyone else, you don't need the name of the syscalls to be able to use this dataset. Just assume that 377 is a name of a feature (just like you could have used the name of its process) or say it means systcall for process X and proceed with your research. at the end of the day, I don't think your machine learning or classification technique will need any of the syscall names to make predictions unless you are doing some sought of text-based mining.

Hi

Can you send the labels of ADFA-LD dataset?

[email protected]

thanks.

Hi Hao,

You should look at the thread in this question. In one of the comments, I explained how to extract the dataset. The reason why you don't need a label. Btw, if by label you mean class label, then remember that the files are organized based on the attacks and the normal folders. So every file in normal folder refers to the normal instances, and the ones in the different attack folders belong to attacks. You can just loop through the folder using your favorite programming language. But if you are referring to the attribute names, then I have explained this in one of the previous comments. 

Hi Hao,

If you do insists on arranging them to make an equal number of columns or features, you need to search through the entire files and extract the number in all or them, then get the unique numbers. I can't remember how many unique numbers were in the entire file. Then, for each file or line, count the frequency of each number, and put it in its respective column. Example say you find out that there are 5 unique number i.e., 1,2,3,4,5, and for the first file there is 1, 1,2,2,2,3 then in csv file of equal dimensions, the first instance in it would be 2,3,1,0,0 because "1" occurs twice, "2" occurs three times, and "3" occurs once, then "4" and "5" occurred zero times. Then you go through the entire folder doing this. This will create equal columns. But not that this will be a very large number of columns for the ADFA dataset. So you might wanna use PCA to do dimensionality reduction.

Hi Obinna,

for my master thesis I 'm working on a research work which analyses the ADFA-LD dataset. I googled it, and found your question here, and I followed the comments, started from two years ago until now. I am really thankful for your interesting and helpful information.

Now coming to my question, for my research I have to parse this dataset and analyse it using various data mining approaches. However, I don't know, what the files stand for in the dataset. The sys call traces are saved in various files, like UTD-0029.txt. But, is each file regarding a specific process or program, or just they are contiguous sys call traces of all.

the next Q is, do you know, other research works that analysed this dataset using machine learning algorithms.

The files having the UTD-*.txt type all belong to the normal class. Each sys call trace is a file of its own. E.g., UTD-0029.txt contain one single trace which in turn contains multiple syscalls. You need to read the paper from the generators of the dataset to understand more about it: http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6555301.

On your second question, the paper they wrote above tried to evaluate it with other ML algorithms. I had a project that was supposed to use this dataset 2years back, but I later abandoned it. But feel free to check out my profile if any of my papers are relevant to your research. Hopefully soon enough I might revisit this or at least write a small conference paper on different ways to extract this dataset. Thanks.

Hi Obinna,

Thank you very much for this thread. It helped me a lot with understanding the ADFA-LD.

To extend Mr. Samimi's question, I'd like to know what does each file represent:

1) Does each file in the training and validation (starting with 'UDT' or 'UDV') represent a single process? If so, is it possible to tell which process it was taken from?

2) Does each file in the attack folders (starting with 'UAD') represent a single process? If so, do you have labels for which processes were attacked?

Hello Obinna Igbe ,

Can you please share the ADFA-LD and ADFA-WD dataset with me ?

it is really helpful if you give it through an email

my mail id: [email protected]

Thanks.

Attached is the label for the data. It is extracted based on Obinna Igbe 's explanation.

Thanks Farouq Muhammad Aliyu

Do you have any useful resources related to Windows Dataset for malware detection.