Thanks, Lennart and Kambiz, for your recommendations on the commercial FE software. I know these software are powerful on nonlinear analysis. Currently I use an in-house code of XFEM for failure analysis of composites. When the material is softening and has a negative stiffness, I am implementing the analysis with a in-house nonlinear solver but find it is hard to get converged results. So I want to try some other available nonlinear solvers which can be integrated into the in-house code.
As Lennart said, there is software (which I would rather refer to as the environment) for making your own programs. They contain libraries and solvers to formulate and then to solve the equations describing some physical phenomena. Apart from the commercial software such as Matlab you can use a free alternative GNU Octave.
If you want the whole open source multi-physics software package I can recommend you Cast3m (http://www-cast3m.cea.fr). I know it for modelling of concrete but I've seen its use in modelling of composite materials, too. The package is huge so it will probably take you time to investigate its possibilities but I think it is very flexible, robust and complete so this should be worth it.
Try OOFEM (http://www.oofem.org/en/oofem.html), Calculix ( http://www.calculix.de/ ) or Elmer (http://www.csc.fi/english/pages/elmer)
They all should have source available. Check Elmer's solver manual for details on their non-linear solvers. http://www.nic.funet.fi/pub/sci/physics/elmer/doc/ElmerSolverManual.pdf
I agree with Marcus. Cast3m and Code_Aster are similar packages which were being developed in parallel in France by CEA and EDF, respectively, both from the field of nuclear engineering. Both are written in Fortran and may serve for similar purposes.
Just to clarify - SALOME is used only for preparation of data and visualisation of results (pre- and post-processing). It was designed as a user interface for Code_Aster. SALOME is based on ParaView, also recommendation-worth open source software.
Depending upon the nature of the problem.What exactly is the analysis type? Ansys Workbench is also providing strong inbuilt packages & codes in its scripting language..
I confirm what has been said about Cast3M and Code_Aster if you need an implicit code, they may be the best choice in general.
Cast3M can be easier for "normal" use and development. But Code_Aster has more options/elements/methods, a wide documentation, and it uses Python scripting language. There is also the fluid equivalent: Code_Saturne.
Depending on what you need exactly (solid/fluid, implicit/explicit, etc.), FreeFem++ can be also very interesting, as well as Elmer, Calculix, OpenFOAM, Impact...
Concerning pre/post-processing, as it has been said, you can use Salome platform, ParaView module, or even directly tools such as Open CASCADE. There is also Gmsh that can be very useful to draw home-made code results.
Finally, for the operating system, CAELinux is a Linux distribution based on Ubuntu and including many interesting FE development tools such as some of the previous ones: Code_Aster, Code_Saturne, Gibi (mesher part of Cast3M), Salome, Gmsh, etc.
If you are looking to learn how to write different types of FEA code, then the text book "Programming the Finite Element Method", 5th Edition, 2014 is a good starting place for a broad range of algorithms. It comes with 70 short driver programs (1-2 pages) for a range of engineering problems, has a library of subroutines and covers parallel processing. The philosophy makes customisation easy as it is not a single monolithic code base.
You could request a copy via your University Library. It is probably useful to have access to a copy even if you choose to adopt one of the other open source software packages mentioned here.
Non-linearity is a very broad term. In the context of FEM non linearity may refer to non-linearity in material properties or in the governing equation. That related to material properties is easy to tackle and most of the FEM solvers (open source or not) do have provision to solve it. The non-linearity of the governing equation is tackled in a unique way for each problem type. One good open source package is Elmer (http://www.csc.fi/elmer) with a variety of solvers (linear, non-linear) for many physical situations (fluid mechanics, magneto dynamics, heat flow and many more). It is a huge code and source is available (requires patience to go through it). However, if solving a particular type of problem in any of these areas is concerned, knowing the source code is not essential and users manual will serve the purpose. In a nutshell, it is worth exploring Elmer for your requirement.