OFFICIAL STATISTICS will continue to serve and function as a valid source. The so-called independent sources will always remain an alternative source on the side way. The state has the resources and verification process. While some states may doctor their statistics, it is the job of the indeoendent data sources to serve its check-and-balance role. Official and unofficial sources will exist side by side. What is not state or non-official will remain unofficial no matter how sophisticated the technology is employed in gathering the data. At the end, the user of statistics will compare the offi ial and unofficial side by side, this will keep the state from over-stating its own numbers, i.e. GDP, employment, etc.

I agree with Professor Paul Louangrath. In addition, I would like to highlight the fact that, despite some governments’ interference with statistic administration, others manage to maintain the independence of their statistic agency. In a number of countries, the independence of the statistic agency is comparable to the central bank's. Moreover, these official statistic agencies keep improving, that they also use what so called smart statistics technologies. Accordingly, these official statistics are on a par with the independent sources which, as mentioned earlier would still be needed for check-and-balance. Last but not least, these official statistics would remain as the sources for international institutions, such as the World Bank, IMF, Comtrade etc.

Official Statistics should be managed by smart and knowledgeable Statisticians.

There will be a dynamic and efficient merger of both techniques; the policy-making process will combine a-posteriori and a-priori statistical learning methods, i.e. statistics will become more creative, constructive and intuitive, in methodical terms.

The official statistics remain even in the presence of smart statistics. Smart statistics do not cover all areas where official statistics are used.

even in the era of smart statistics there will be still need for an official authority to confirm and demarcate the correct figures from incorrect.

Peter -

I worked at a government statistical agency for energy statistics, and when I retired in 2014 they had already turned a lot of attention to the kind of data you might mean, such as hourly data that might be available on electric power transmission, or daily gas price data, both from third party sources. There seemed to be a frenzy over quick, highly detailed data. But like the frenzy over big data, some of it is promising, and some of it is not going to meet expectations. But this isn't the first time that something promising then becomes somewhat disappointing.

The purpose of official statistics may more likely be for planning and for economic decision making, as opposed to engineering uses or special, temporary conditions.

Official statistics are generally at a more aggregate and more organized level. Some of the lower level data may not be available, or may not be releasable, or just not relevant to the level of information needed for certain decision makers. Sampling is generally needed. The following invited presentation is what I gave in the fall of 2017 when I was asked to address a lunch group of mathematical statisticians at the agency from which I had retired:

https://www.researchgate.net/publication/319914742_Quasi-Cutoff_Sampling_and_the_Classical_Ratio_Estimator_-_Application_to_Establishment_Surveys_for_Official_Statistics_at_the_US_Energy_Information_Administration_-_Historical_Development

An agency should keep up, not only with changing technology, but also the changing needs of the public and often a large number of interests. To some people certain data may be extremely important, but to others they cannot see that at all, and what is important to them is something entirely different. The age of information and the age of technology have much in common, but official statistics are very important for economic and social issues with more of a "big picture" approach.

I can also think of a highly technical approach which incorporates survey sampling and estimation (design-based)/prediction (model-based): remote sensing for forestry information.

One can be excited about new technology, but just because something is old does not mean it is not needed. Also, it may take a good deal of development before major improvements in a technique are realized.

Finally there is analysis. I do not think that is the major job of a statistical agency supplying official statistics. The major job is to provide good data for others to use. But some time series information may be provided, and feature articles can be used to study and present information on issues of interest.

For Official Statistics, tables of aggregate level data and information on accuracy, such as relative standard errors, and perhaps even descriptions of factors influencing data collection, can be helpful. Metadata may be considered.

Best wishes - Jim

PS - Following is one of my projects on ResearchGate:

https://www.researchgate.net/project/Cutoff-and-quasi-cutoff-sampling-with-prediction-for-Official-Statistics

Thanks, Jim, for this great view on the responsibility of official statistics for data, the modes of collection and their use. I think that official statistics should play a stronger role in monitoring infection testing and in supporting the development of tools to use mobil positioning or related data for diagnosing contacts of individuals. Much of what you are saying applies to these points. Statistics should also be involved in decisions based on such data by experts and politics.

Kind regards, Peter

For the 2022 Joint Statistical Meetings (JSM), earlier this month, I gave a poster presentation regarding how the successful use of quasi-cutoff sampling with prediction for energy establishment survey data over the last 32 years and counting, for tens of thousands of aggregations (probably a low estimate), might be useful for other subject matter applications.

In June I sent the poster for this to the Chief Statistician of Canada, noting its potential use for their Miller's Surveys.

The Assistant Chief Statistician for Economic Statistics wrote back, indicating interest and stating that he further disseminated the poster.

For the Miller’s Surveys one could make use of the data, or at least the survey forms, which are already being collected. 

Here is the poster: 

https://www.researchgate.net/publication/361024156_Poster_for_Application_of_Efficient_Sampling_with_Prediction_for_Skewed_Data

The paper may now be found at

https://www.researchgate.net/publication/362370770_Application_of_Efficient_Sampling_with_Prediction_for_Skewed_Data 

Some repetition is to be found between sections which may be helpful as this document may be useful as a guide for applications, and it has been observed that this may increase clarity. 

Note that US EPA Toxic Release Inventory data are used in an example in the paper.

Cheers - Jim 

Both the environment in which agencies operate, and the demands made for official statistics in policy are changing in unpredictable ways. Responses refer to the continuing mission of official statistics - hardly changed - and the pressures on the methods community to deliver products that are both relevant and authoritative, and I would add sufficient to meet policy demands, not only of national governments but across governments. There are indeed emerging tools, as Dr Knaub has indicated, that can guide future investment by agencies, balanced between design-based surveys and transaction-based quasi-continuous exhaust datasets. The full potential of these methods remains to be demonstrated, but the theory has been advanced and is being tested on real applications.

With all of the changes in needs and available data, I think we will still see that for monitoring the economy, there may always be a need to obtain aggregate level information from establishment surveys on products. At the US Energy Information Administration, energy market information has been collected continuously for decades, and the need for this information will persist. I have noted to some people that quasi-cutoff sampling has "... proven highly effective at the US Energy Information Administration (EIA), using ratio modeling for inference. (Example application: retail electric sales volume by economic end-use sector and State.)" See

https://www.researchgate.net/publication/374294842_Stratification_and_other_Issues_for_Best_US_Federal_Establishment_Survey_Quasi-Cutoff_Sampling

There can be changes and shifts in application, and technology may alter techniques used, but I expect that establishment surveys will likely be needed for some purposes for the foreseeable future.

Is there will be any specialisations to treat the data or sample including engineering , environmental or biological , artifial intelligence or office maintainance w.r.t SQC or R- Analysis etc ?.

https://cros-legacy.ec.europa.eu/content/trusted-smart-statistics-nutshell_en