In economic point of view, how to proceed if i'm interested to study the existing structure of an institutional system, to identify constraints, and improve efficiency? Any basic tools, methods in it?
You can start from identifying the main end-products delivered by that system, then moving backward continue by following the process. In doing this, you can map the responsibility areas involved in that process (by performing some task) and the resources they use. once you did so, you can start understanding how efficiently or effectively the system is the system is performing against a (competitors, expected, desired)
I second the question in a slightly different way. Is there any method to evaluate the performance of institutions? Or more generally, how can we evaluate the governance efficiency within a country, a city, or even a neighborhood?
formal economic methods usually refer to relative efficiency by comparing several entities (companies, institutions, regions, countries) called 'decision making units' (DMUs). First step is to define 1) production function with inputs and outputs of the process and 2) DMUs that have a control over the process. With these defined one can apply parametric (e.g. stochastic frontier analysis) or non-parametric (e.g. data envelopment analysis) methods to obtain numerical values of efficiency (either technical or allocative).
This approach does not allow to assess the efficiency of a single unit; numerous units are needed.
I recommend Coelli et al. "An introduction to efficiency and productivity analysis" as a very good introductory textbook.
You might need to apply Institutional Analysis and Development (IAD) Framework developed by Elinor Ostrom, a prominent scholar from Indiana University who was awarded Nobel Prize in economics in 2009. This framework will be very much useful to address complexity in institution-related issues.
The IAD Framewok was later expanded by Floriane Clement by adding more exogenous variables to incorporate political context and distribution of power amongst actors.