@David is correct, with a given policy, we need to define the output that we want to measure the effectiveness. "Effectiveness" itself must be a fixed standard or guideline against which the assessment of the observed values is made.
DEFINE EFFECTIVENESS: Now suppose we define that the target object of measurement is OUTPUT with a threshold target of X; thus if O > X then it is categorized as effective and if O < X then it is ineffective. Suppose further that the policy (π) is implemented at a fixed and unique time t. In order for the conclude whether the policy is effective or not, we need to allow the policy to run over time. The policy in this case is treated as a stimulus. The measurement is to measure the system response to the stimulus. Below are two possible approaches---there could be other methods as well.
POISSON DISTRIBUTION FOR BINOMIAL COUNTS: The length of time that we measure at t1 and t2 may not be equi-distance, i.e. one measurement in 3 months after implementation and the second measurement in 6 months after the first one. For each count at any period, let N be the total number of counts that is classified as effectiveness. The average rate for the two counts are: R1 = N1/t1 and R2 = N2/t2. Under this scenario, we would have a 2 counts of Poisson distribution. The test for significance is given by:
(1) Z = (R1 - R2) / sqrt((R1/t1 + (R2/t2))
If Zobs is significant then we would assert that the claimed effectiveness is statistically significant.
POLLING APPROACH: Using the same definition above: effectiveness = yes and ineffectiveness = no. we would still polling (taking data reading) in two time periods, but disregard the length of time lapsing between the first poll and second poll. Let's put the effectiveness and effectiveness dichotomy into a 2 X 2 table:
_____________________
YES1 NO1
YES2 a b
NO2 c d
_____________________
Column = first poll
Row = second poll
After defining YES = effectiveness, the test statistic is given by:
(2.2) N = (a + c) + (b + d) (a + b) + (c + d) = total counts.
McNemar provided an alternative argument using chi-square test:
(3) X2 = (b + c)2 / b + c
REFERENCES:
McNemar, Quinn (June 18, 1947). “Note on the sampling error of the difference between correlated proportions or percentages.” Psychometrika 12 (2): 153–157.