if the policy is focused on entrepreneurship, one of the main measures is the number of companies created , but also is very important to measure the sustainability of these what could be analyzed incorporating measures of profitability.

This is very hard.

I would recommend reading Realistic Evaluation by Pawson and Tilley (SAGE)

A difficult task. You would not only have to identify specific factors to measure and compare before/after the policy. But you would also have to take not of external factors beyond policy impacting entrepreneurial activities and differentiate what has been triggered by the policy, what by other factors (economic overall growth, foreign demand, etc). In other words, it will be very difficult to make a compelling case.

A different approach could be qualitative expert interviews, asking those how designed, implemented, and evaluated the policies as well as those who benefited from them.

I agree with Arne and I recommend key informant interviewing as a realistic way forward

Policy is a term that refers to many areas of development such as economic, social, security, employment, social, defence and so on. So there are lots of indicators that can measure policies. Policies, in the first place, must be implemented. They are implemented through various strategies and action plans. Some policies are long-term in nature while others may be short term. Some can be measured directly while others may need proxy indicators. You may need to set a benchmark time or period to see if policy implementations are impactful and of course there must be a target point.

Lets get one example of a policy out of this. Say security policy. First .you have to define clearly what security policy you are trying to implement. Is it social security, is it national security, is it defense security or is it cyber security? After defining the type of security you are involved with then it is important to list out the implementation plans ranging from strategies to action plans. For each of the strategies and action plans then there is a need to set benchmark and target years for the implementation. And finally, the measurement - development indicators to measure success or otherwise.

If for example, you are talking about social security then various indicators or measurement metrics can be formulated. One of them is the level of poverty. The metrics - the number or percentage of population that fall above poverty line or minimum cost of basic needs, number of population released from unemployment benefits, job creation or number of new employment, etc. Quite a lot of them to measure the successful implementation of a social security policy.

But most important, measurements or indicators must reflect the policies being implemented. Of course, there are other aspects that need to be considered apart from those that I have explained above such as the implementation agencies, coordination, resources that are needed, etc.

Thanks Mahindra Wijesiri for the compliment.

Actually i did not read the longer version of the question. I responded to the shorter version "How to measure the impact of a policy?"

Specifically to entrepreneurship policy then the policy objectives need to be examined as well. Entrepreneurship (development) is rather wide in nature. As we see around the world, different countries have different stages of their entrepreneurship development and different types of entrepreneurs that they sought to develop further. There are countries that are already well advanced with their communities of entrepreneurs while others sought for specific types such as IT-preneurs. The general ideas however are the same as in my previous answers.

In the case of a more general entrepreneurial development policy some basic indicators of success include the number of new entrepreneurs being developed, the value-added generated by the new entrepreneurs as percentage of GDP, the number of employment generated, the amount of capital investment, etc. However, one important element of success is their sustainability and growth. How many have emerged successfully out of start-ups comfort.  You may want to monitor these over a longer period of time.

In some countries, entrepreneurial development may also involve government financing and grants and as well as the involvement of venture capitalists or angel investors. Entrepreneurship, therefore, is also a question of independence and ability to repay these investors.

But entrepreneurship culture is not just born out of nothing or automatic. Education and skills development play important role right from the early age. It is now widely accepted that entrepreneurship can be taught and learnt. A person can be motivated to become an entrepreneur through proper education and coaching.  What about others that help develop and encourage entrepreneurship or what is often referred to entrepreneurial eco-system? How influential are they in generating more people to go into entrepreneurship? So, these are also some of the things that need to be considered as well.

Thankyou All for your valuable input. The suggestions, views all helped a lot in understanding the measure of Policy to a certain extent. Appreciate your help.

Dear Adnan,

I suggest you to consider the "Six Steps to Heaven", a classic by David Storey. Here below, I give you links toward a free OECD doc and the WP:

http://www.oecd.org/cfe/smes/31919294.pdf

https://www2.warwick.ac.uk/fac/soc/wbs/research/ei/research/working_papers/wp59-six_steps.pdf

Best,

Marcus

To my astonishment (and speaking as a a natural scientist)  most conventional economists who are expecting /requesting that their economic policies be put into practice do not put their policy concepts to an empirical or scientific text.  If indeed economists wish or expect their policies implemented then it seems to me that they should:  1) state their case for the expected benefits of a given economic policy as an hypothesis or hypotheses; for example if policy "A" is implemented then its efficacy will be judged as positive (or, technically, consistent with the hypothesis) if outcome "B" occurs.  this should be done before implementation.  Subsequently, if B does not occur then there is no support that the policy achieves its intended benefits.   For example during the 1990s there were great efforts to institute the "Washington consensus", a set of neoliberal policies, in various countries in Latin America.    The results of this policy, rarely formally tested or even examined,  were, when presented as hypotheses , rarely supported (E.g. Kroeger and Montanye 2000 and references therein.   Additionally they caused great social and economic harm in other ways.

For a second example, in the United States more recently we often hear that private companies are more "efficient" than governmental one.  June Sekara developed this as an hypothesis and tested it explicitly, and found the hypothesis not supported at all. 

For a third example, Hallock et al. tested through formal quantitative models the hypothesis that nation-by-nation oil reserves were at the high (vs low) end of various estimates.  They returned 10 years later to test their models and found for the large majority of countries that despite increases in prices oil production tended to follow (a Hubbert curve with) the low rather than high (or even medium) estimates.   My point is that although often testing the implications of policy is difficult it can be done and should be tried.   There is no reason that we should not bring economics as a discipline out of the realm of abstract theory and expert guess and into the world of real science.

references: 

• Montanye, Dawn R. MS; 1998.  Examining Sustainability: An Evaluation of USAID Policies for Agricultural Export-Led Growth in Costa Rica  (summarized in Kroeger, T. and Montanye D. 2000 An assessment of structural adjustment policies in Costa RIca.  p. 665-694 in C. Hall (ed.). Quantifying sustainable development: The future of tropical Economies. Academic Press, San Diego.

.      Hallock Jr., John L., Wei Wu, Charles A.S. Hall, Michael Jefferson. 2014.  Forecasting the limits to the availability and diversity of global conventional oil supply:

Validation.  Energy 64: 130-153.   (Article 12 in:             http://www.sciencedirect.com/science/journal/03605442/64/supp/C     

http://www.sciencedirect.com/science/journal/03605442/open-access

Sekara, June.   GLOBAL DEVELOPMENT AND ENVIRONMENT INSTITUTE

WORKING PAPER NO. 15-02  Economics and the Near-Death Experience

of Democratic Governance May 2015  (Soon to be published in "Economic Briefs" Springer. 

Adan - Great question. In most policy arenas, it is a challenge to isolate particular policy initiatives and their impact; it is also difficult to measure only the intended benefit.  Some of the best approaches seem not to consider the important points made by Professor Hall. A recommended alternative would be to examine the broader domain the policy choice represents and to assess outcomes in a number of related areas; in addition, there is more that one kind of stakeholder (and related measures) and one might consider various kinds of stakeholders and their interests.  See the chapter on performance in  Koliba et. al. (2011) Governance Networks in Public Administration and Public Policy; see also

H. George Frederickson and Jack W. Meek (2011) Bureaucratie sans Frontiéres: Legitimacy, Authority, Accountability in Geo-Governance Systems, in Edoardo Ongaro, Andrew Massey, Marc Holzer, Ellen Wayenberg (eds.) Policy, Performance and Management in governance and Intergovernmental Relations: Transatlantic Perspectives. Cheltenham, UK: Edward Elgar.

Firstly a policy has an ideological framework that is used to provide alternatives that could address the issues at hand. Generally, policies operate within a systems environment that could be dynamic and or complex therefore requiring a keen focus on cause and effect.

My view is to identify any particular trend or behaviour that has continued and/ or developed subsequent to the implementation of the policy and evaluate the cause and effect of such trends.

In impact evaluations it is also important to have accurate baseline data, for example the number of new start-ups in the different sectors of the economy. (Wile taking cognisance that the extraneous factors could also affect the development of entrepreneurship. Developing and clarifying causality for entrepreneurship is therefore crtical.

Perhaps to answer that particular question, before  we  have to ask if we can measure, something done without clear objectives away from any forecast measurability. The political objectives are not clear or not interested to know what is shameful and unacceptable or the fact that there is no attitude or ability to set such targets.

A 2010 work exposes the difficulty of evaluating the performance of government, different criteria (effectiveness, efficiency, democracy ...) are used; different indicators are used also by evaluators (ECB,  IMF, ...) or some of the indicators are not objective or reliable (perception of some actors chosen)

We have to wonder seriously whether the political system is in practice an obstacle to a scientific governance. The rules of governance framework does not fit with an inductive theory from observation. What to do to not follow from the fourth century BC well?

Vine. Ballart, X, (2010) "A difficult prey to catch: the performance of governments and public administrations.". Spanish Journal of Political Science No. 22.

The impact study may help you to answer your Question. for instance, change in bank rate policy has its impact on lending rates and deposit rates: before and after change in the value of these variable help to measure the impact of changed policy.

I guess, the simplest way is to compare Doing Business indicators (time spent for and costs of typical business transactions) before new Entrepreneurial Policy implementation had started and since it had started. Economic Freedom in the world indicators are much  more subjective but provide useful framework for your analysis (like Protection of property rights, impartial courts etc). Foreign Direct Investment (FDI) and Small and Medium Enterprises (SME) employment dynamics (before and after) are indirect measures and a little bit to general, but if you will detect significant increase in SME employment and in FDI with 1-3 years lags,  the increase could be explained by the new policies impact as well. 

Considero que las políticas públicas pretenden, entre otros propósitos, construir nuevas instituciones, es decir, reglas de juego formales, que regulan derechos de propiedad, introducción de correctivos económicos o de mercado, nuevas reglas de cooperación entre el gobierno y los particulares. Por supuesto todo lo anterior se plantea en un contexto específico, para una realidad concreta que se quiere controlar o transformar. Si esto es así, entonces no se visualizan fórmulas de validez universal. Pues cada problema social, cultural, medioambiental, económico o político requiere un método a la medida, el cual tendrá siempre el color de la imperfección propio de la construcción humana.  

Dear Adnan

I guess that you can do by one of the following methods

1. Experiment study

2. The impact study

Best

Canh

I think

1. you can also use CGE if policy (though specifically for entrepreneurs) but is expected to affect the whole economy.

2. As mentioned above you can define two groups control and treated(who opted This policies). or same group but use data before and after the implementation of policy. You have to collect primary data.

Thankyou Rizwana and Nguyen Phuc Canh for your suggestions

Adnan,

I am going through a similar exercise.  The policies to be assessed are in the area of productivity.  A triangulated approach will be adopted:

a. Have our Productivity Instutute take a researched assessment and furnish their best evidence-based judgement on each of the various productivity policies

b. Convene an expert focus group of leaders in business, employee associations, civil society, academia etc to give their expert opinion

c. Conduct a general poll to get the stakeholders' opinion of impact of each of the policies.

https://www.researchgate.net/publication/299747565_Towards_a_talent-driven_outward-oriented_globally-competitive_SME_framework_Discussion_Paper

https://www.researchgate.net/publication/305687139_Priority_policy_recommendations_for_transforming_individual_productivity_and_SME_competitiveness_in_Jamaica

Technical Report Towards a talent-driven outward-oriented globally-competitiv...

Technical Report Priority policy recommendations for transforming individual ...

Adnan,

See the links bellow for examples of quantitative policy impact measurement:

Best regards

http://www.iacajournal.org/articles/abstract/10.18352/ijca.180/

http://www.iacajournal.org/articles/abstract/10.18352/ijca.215/

Thank you Pedro

Two years later, how did you do? Do you have a paper for us? Otherwise, the key to robust policy evaluation is framing. Sketch out the forces at play. Physics will tell you it is a dynamical system. At that point you will throw out determinism. If the phenomenon is non-ergodic (see paper below), you might have to accept parametric mapping as the end product - show how the parameters have changed. Here is the non-ergodicity paper: Article Public-Private Partnerships (PPP) on Moulding State Structur...