No, it is not necessarily unethical, in my experience, especially when you are building on your previous work, and especially if you work in a unique area. References should not just be window dressing. They should be relevant. Ideally you found them as you did research before you wrote your paper, although some may come in along the way.
More references from other people, however, may be taken as a sign that you did properly research your topic.
Self-citation is considered acceptable in the academic community when used appropriately and relevantly. While it allows authors to build on previous work, excessive self-citation for the purpose of inflating citation counts can raise ethical concerns. You can find more information on self-citation and its impact on research integrity on our website: https://researchbrains.com/self-citation/.