The dataset and algorithm should i use in the network anomaly detection using artificial intelligence are

Different Types of Anomalies:

• Point anomalies – if a data point is too far from the rest, it falls into the category of point anomalies. The above example of bank transaction illustrates point anomalies.
• Collective anomalies. The collective anomaly denotes a collection of anomalous with respect to the whole dataset, but not individual objects. Example: breaking rhythm in ECG (Electrocardiogram).

try to Intrusion detection Evaluation data sets

I prefare to make your Own dataset , using different network lead to diffent type of attack, list this attack and each duration, and also chose your attribute.

you also can use

Software : https://github.com/nrajasin/Network-intrusion-dataset-creator

Dataset would be more based on your usecase and your knowledge about that particular domain where you carefully buidl your feature vector.

Probability density funciton would be one choice. If your dataset is huge, then an Auto Encoder. Autoencoder would be trained against a clear dataset, over train it with a dataset having very very little anamolus data, like 10 in million. During testing wth a anamolus input, if the reconstruction error is very high, this would mark a anamolus input.