You can follow this link -

https://datafloq.com/read/5-key-deep-learning-ai-challenges-in-2018/5400

Adding single Class is slow, though it's interesting. You are adding class of same category or different. And how you are proceeding? I mean you are just using train module to same architecture or you also optimizing your architecture?

And also it will be interesting to keep a copy of your train modules plus visualize the parameters. Share it if you can or open a discussion a put your updates if you can.

Now come to your final part of your question, I think once your number of Classes increase or it become more diversified the result will start degrading.

my network is perfect in the performance but I am adding several classes. I am using transfer learning. and the architecture is optimized for the current problem but as we add more classes, I want to check if I need to check anything for the model.

It is public challenge that deep model is suitable to all applications. Deep models aren't always robust to all applications.

I presume you've seen this paper: Article How transferable are features in deep neural networks?

It's relatively old but highlights some problems you might encounter. If any of your new classes are suitably different to your original classes, the network might not be able to retrain efficiently.