I want a function in which R applies the maximum likelihood concept to Separation of the Normally distributed components of size-frequency samples. A similar tool FISAT II for example.
I think that this is what the authors did here: http://www.scielo.cl/scielo.php?script=sci_arttext&pid=S0718-560X2012000200025
They checked FISAT II using R, so it might be worth contacting the authors.
There is the package called fpc which looks like it does what you're looking for: https://cran.r-project.org/web/packages/fpc/fpc.pdf it uses Bhattacharya, but does not have a normsep equivalent.
For your question about a way to categorize the data of each mode, I am not sure but you can check this site: http://tinyheero.github.io/2015/10/13/mixture-model.html. Hope that it will help. Best regards, Katerina