Can a FigTree file is compatible and reproduced in R software? Also, can anyone give me some links or reference for trait mapping in phylogenetic trees?
The best way to answer the question about what kinds of files can be opened in R software is to find the relevant Task View on CRAN. Below I've copied what the Phylogenetics Task View says about reading trees into R. As for ancestral state estimation, the literature is quite large and several of the recent, most flexible methods for quantitative traits, are not implemented in R. A nice recent summary/tutorial that emphasizes methods that are available in R is by Liam Revell, the author of the phytools package: http://www.phytools.org/eqg2015/asr.html
As for getting trees into R, from the Task View:
Getting trees into R : Trees in R are usually stored in the S3 phylo class (implemented in ape), though the S4 phylo4 class (implemented in phylobase) is also available. ape can read trees from external files in newick format (sometimes popularly known as phylip format) or NEXUS format. It can also read trees input by hand as a newick string (i.e., "(human,(chimp,bonobo));"). phylobase and its lighter weight sibling rncl can use the Nexus Class Library to read NEXUS, Newick, and other tree formats. treebase can search for and load trees from the online tree repository TreeBASE, rdryad can pull data from the online data repository Dryad. RNeXML can read, write, and process metadata for the NeXML format. PHYLOCH can load trees from BEAST, MrBayes, and other phylogenetics programs (PHYLOCH is only available from the author's website). phyext2 can read and write various tree formats, including simmap formats. rotl can pull in a synthetic tree and individual study trees from the Open Tree of Life project. The treeio package can read trees in Newick, Nexus, New Hampshire eXtended format (NHX), jplace and Phylip formats and data output from BEAST, EPA, HyPhy, MrBayes, PAML, PHYLDOG, pplacer, r8s, RAxML and RevBayes. phylogram can convert Newick files into dendrogram objects.
If I understand you well, I think you can use the R package phytools (https://cran.r-project.org/web/packages/phytools/phytools.pdf) it can load neweek and nexus files, which are the standard in FigTree...
To map the traits in your tree you can use Mesquite (https://www.mesquiteproject.org).
R is definitely the way to go for the most flexibility. However, other programs exist for mapping traits like RASP (https://www.sciencedirect.com/science/article/pii/S1055790315000652) and Mesquite. You should consider reading Luke Harmon’s book on modern phylogenetic comparative analyses.