I think it is quite helpful to have an idea about librarianship and the workflow in a library, to have background knowledge in information retrieval and IT, in Busisness and Marketing.
I could imagine working as a Subject Librarian without that training, but this wouldnt help to understand the library context.
The philosophy of service varies from institution to institution so there needs to be at least some basic understanding of the values and of the parameters within which the librarian functions. IMHO subject librarians should be a resource for the library as well as the campus and available to help with language and subject help to other departments such as cataloging and resource sharing.
This might be necessary to save time and man hour as previous knowledge in a particular area, field or discipline advances such knowledge and also this depends on the nomenclature of each individual institution, some are subject specialists while others are not
I think there are generic professional skills. Some colleagues may find they lean towards different aspects of the profession. Some are better at managing resources others are better dealing with clients/users. The most recent iteration I know of is the one by CILIP (UK) [ http://www.cilip.org.uk/jobs-careers/professional-knowledge-and-skills-base/What-is-in-the-PKSB/Pages/Whatiscovered.aspx ].
The secret of providing a good service is to have the insight / vision / flexibility to match skills in the context in which they are to be delivered. A Subject Librarian should master the literatures of their subject, a Health Service Librarian should have a knowledge of the health literature and the contexts in which it is applied. I suggest its a combination of two things: 1) Learning the Literature Structures (e.g. core journals and databases) and 2) experience and practice make for good professional practice.
I should add that a dominant paradigm from one sector might dominate professional thinking. I would argue that it is possible to train specialists in Academic or Health Librarianship as part of the library curricula to better prepare people for the sectors in which most will work. The very act of thinking about this raises the level of debate.
I don't think you need to specifically train a person as a subject librarain, as long as that person has a good background in the specific subject matter and attends focused conferences to keep updated on new directions in that subject and new technology as it evolves. However, if they lack instructional skills, that would be an important area for training.
As the mental and behavioral health librarian at my university, those questions are referred to me. I continue to learn in this field, not only to help students find the resources they need for their coursework, but also to furnish high-quality literatrue reviews for faculty, to upgrade reading lists for courses, and to contribute to the professional literature.