The first issue with any public-use file (open data) is providing a file that allows reproduction of 1-2 (but hopefully more) analyses that might be performed the the original, non-public file prior to the 'masking' of certain fields to prevent re-identification. If the file has valid analytic properties, then the data producer should (attempt to) justify that re-identification of a small proportion of individuals is exceptionally difficult or impossible.

 This has been an open area of research for 35+ years. Do a bunch of tabulations (queries) with the underlying microdata in a manner so that individuals cannot reconstruct the underlying microdata from the tabulations (even when noise (epsilon) is added in a suitable manner). At one point, major database groups and IBM concluded that it was an impossible problem (even after millions in dollars of NSF support).

 In the mid 1990s, Latanya Sweeney (then a CS Ph.D. student at MIT, now a Harvard professor) took an 'anonymized' set of health data from Massachusetts State employees and showed how to re-identify most of them using a Massachusetts voter registration database. The health data had been anonymized by removing individual's names, SSNs, heath insurance IDs, doctor's names, hospital names and just about any other identifiers that individuals could think of. For analytic purposes, ZIP codes, sex, and date-of-birth were left in the file and these fields were used to re-identify more than 70% of the individuals in the file, including the Governor.

 In 2003 Dinur and Nissim provided methods with rigorous guarantees on the privacy.

Dinur, I., and Nissim, K. (2003), “Revealing Information while Preserving Privacy,” ACM PODS Conference, 202-210.

Much of the work in recent years has been on improving analytic properties.

Dwork, C. (2008), “Differential Privacy: A Survey of Results,” in (M. Agrawal et al., eds.) TAMC 2008, LNCS 4978, 1-19.

Barak, B., Chaudhuri, K., Dwork, C., Kale, S., McSherry, F., and Talwar, K. (2007), “Privacy, Accuracy, and Consistency Too: A Holistic Solution to Contingency Table Release,” PODS ’07, Beijing, China.

Hardt, M,, Ligget, K., and McSherry, F. A (2010), A simple and practical algorithm for differentially private data release, available at http://arxiv.org/abs/1012.4763 .

Over the last seven years, the American Statistical Association has had three invited paper sessions at their Annual Meeting on differential privacy. Different groups would like to provide 'protected' microdata with guarantees on privacy and on valid analytic properties.

The answer is that there are presently no systematic methods of both assuring analytic properties and privacy. As long as your original data are suitably clean (this is not always assured in many databases), then your tabulations should be valid. Unfortunately, there are very sophisticated methods of working backwards from the tabulations to subsets of the microdata. Look at the following for improved methods of working backwards from the tabulations to the microdata.

Dwork, C. and Yekhanin, S. (2008), “New Efficient Attacks on Statistical Disclosure Control Mechanisms,” Advances in Cryptology—CRYPTO 2008, to appear, also at http://research.microsoft.com/research/sv/DatabasePrivacy/dy08.pdf .

Many CS and other researchers have shown how to re-identify with seemingly innocuous files.  The used public-use IMDB information to re-identify individuals in the Netflix public-use that Netflix subsequently took down.

Narayanan, A. and Shmatikov, V. (2008), Robust De-anonymization of Large Sparse Datasets, Proceedings of the 2008 IEEE Symposium on Security and Privacy, 111-125.

The first issue with any public-use file (open data) is providing a file that allows reproduction of 1-2 (but hopefully more) analyses that might be performed the the original, non-public file prior to the 'masking' of certain fields to prevent re-identification. If the file has valid analytic properties, then the data producer should (attempt to) justify that re-identification of a small proportion of individuals is exceptionally difficult or impossible.

 This has been an open area of research for 35+ years. Do a bunch of tabulations (queries) with the underlying microdata in a manner so that individuals cannot reconstruct the underlying microdata from the tabulations (even when noise (epsilon) is added in a suitable manner). At one point, major database groups and IBM concluded that it was an impossible problem (even after millions in dollars of NSF support).

 In the mid 1990s, Latanya Sweeney (then a CS Ph.D. student at MIT, now a Harvard professor) took an 'anonymized' set of health data from Massachusetts State employees and showed how to re-identify most of them using a Massachusetts voter registration database. The health data had been anonymized by removing individual's names, SSNs, heath insurance IDs, doctor's names, hospital names and just about any other identifiers that individuals could think of. For analytic purposes, ZIP codes, sex, and date-of-birth were left in the file and these fields were used to re-identify more than 70% of the individuals in the file, including the Governor.

 In 2003 Dinur and Nissim provided methods with rigorous guarantees on the privacy.

Dinur, I., and Nissim, K. (2003), “Revealing Information while Preserving Privacy,” ACM PODS Conference, 202-210.

Much of the work in recent years has been on improving analytic properties.

Dwork, C. (2008), “Differential Privacy: A Survey of Results,” in (M. Agrawal et al., eds.) TAMC 2008, LNCS 4978, 1-19.

Barak, B., Chaudhuri, K., Dwork, C., Kale, S., McSherry, F., and Talwar, K. (2007), “Privacy, Accuracy, and Consistency Too: A Holistic Solution to Contingency Table Release,” PODS ’07, Beijing, China.

Hardt, M,, Ligget, K., and McSherry, F. A (2010), A simple and practical algorithm for differentially private data release, available at http://arxiv.org/abs/1012.4763 .

Over the last seven years, the American Statistical Association has had three invited paper sessions at their Annual Meeting on differential privacy. Different groups would like to provide 'protected' microdata with guarantees on privacy and on valid analytic properties.

The answer is that there are presently no systematic methods of both assuring analytic properties and privacy. As long as your original data are suitably clean (this is not always assured in many databases), then your tabulations should be valid. Unfortunately, there are very sophisticated methods of working backwards from the tabulations to subsets of the microdata. Look at the following for improved methods of working backwards from the tabulations to the microdata.

Dwork, C. and Yekhanin, S. (2008), “New Efficient Attacks on Statistical Disclosure Control Mechanisms,” Advances in Cryptology—CRYPTO 2008, to appear, also at http://research.microsoft.com/research/sv/DatabasePrivacy/dy08.pdf .

Many CS and other researchers have shown how to re-identify with seemingly innocuous files.  The used public-use IMDB information to re-identify individuals in the Netflix public-use that Netflix subsequently took down.

Narayanan, A. and Shmatikov, V. (2008), Robust De-anonymization of Large Sparse Datasets, Proceedings of the 2008 IEEE Symposium on Security and Privacy, 111-125.

While I applaud much of the effort by researchers to arrive at encrypted databases, the problem is (at least to me) not all that difficult from an application standpoint. I have personally developed 4-different encryption schemes, the best being a encryption at the row versus database level. Even if you could decrypt A row, you would not be able to utilize the same algorithm for other rows, since each row encrypts/decrypts independently. The company I developed this for actually patented it (lucky them).  As for medical information, I have done this as well, not only masking information but actually altering information so while it's statistically the same, is modified to mask the true values (you can get the true value back later). To me the real issue is NOT encryption but who knows HOW the encryption/decryption is done. As long as you have people who are in-the-know regarding how this is done, then you have to accept the fact that someone has the keys to the information and there is always a distinct possibility that it will get out.

Joe.  The role of 'open' or 'public-use' data is to provide data to individuals (possibly via a web site) so that individuals can use the data in any statistical package or programs that they would write.  The idea is the 'public-use' data allows individuals to perform analyses that approximate analyses that achieve on the original, more confidential and private data.

Ad hoc methods for blanking out values in records or 'masking' various values in records while presumably preserving some analytic properties were used for more than thirty years.  Over the last few years, computer scientists and some economists have been able to re-identify in some (many?) of the public-use files.

The following is a particularly nice example of how to re-identify by two statisticians using outputs from regression in a public-use file.

Palley, M. A., and Simonoff, J. S. (1987), “The Use of Regression Methodology for the Compromise of Confidential Information in Statistical Databases,” ACM Transactions on Database Systems, 12 (4), 593-608.

The following is a particularly nice paper because it gives so much insight into how the cryptographic methods work.  Sergei Yekhanin is a Russian Number Theorist whose MIT CS Ph.D. thesis won the ACM Best Dissertation award.

Dwork, C. and Yekhanin, S. (2008), “New Efficient Attacks on Statistical Disclosure Control Mechanisms,” Advances in Cryptology—CRYPTO 2008, also at http://research.microsoft.com/research/sv/DatabasePrivacy/dy08.pdf .

Totally agree, but I fail to see what the problem is in creating and making available anonymous database stores. I have done this multiple times and have yet to encounter any issues (at least to my knowledge) of anybody being able to identify any individual whose data I encrypted. 

Joe.  Good comments.  I will try to do additional clarification.  With public-use (open) data individuals try to mask various fields by adding noise so that various aggregates (sufficient statistics) are preserved in a manner that they can be deduced via a non-trivial conversion or models of the overall data are developed that allow draws of 'synthetic records' from the data.  In all situations, the data can be used in ordinary statistical packages.

Agrawal, R., and Srikant, R. (2000), Privacy Preserving Data Mining, Proceedings of the ACM SIGMOD 2000, 439-450.  Here they add noise to streams of data that allow analysis of the data.  Their method significantly reduced the risk of re-identification with procedures known at the time but not with some of the later methods that were developed.

We use encryption in the sense that are harddrives (both PCs and laptops) contain encrypted files.  In certain situations where we can transmit 'secure' data we must encrypt the data and via separate means transmit the keys for the de-encryption.  Under all situations, the data must analyzed only on Census Bureau machines or at certain secure university data centers that are run by sworn Census Bureau employees.  Our security arrangements are reviewed by the NSA (under contract to IRS and SSA) every few years.  Under no circumstances may our private data (Titles 13, 15, and 26) be analyzed on non-secure machines.

Here's the problem: I implemented everything that you described in 1999 and it's still in use (I checked). The difference in your comments and mine is that I was presented with a problem and solved it, because there was a need to solve it, while many researchers talk about the problem rather than solve it (personal experience working at a university and one of the reasons I left).

What do you think about the open source tools of data anonymous? Are they powerful and reliable? Is there any ranking about the data anonymous tools?

Unfortunately, I cannot respond to this question, since I wrote my own and never was presented with the issue since 2000. 

Hello Vahid,

I came across few methods of privacy preserving big data publishing such as, MapReduce based Anonymization, A hybrid approach for scalable sub-tree anonymization over big data using MapReduce on cloud, Airavat, and A Scalable Two-Phase Top-Down Specialization Approach for Data Anonymization Using MapReduce on Cloud. I am also working on one of my own approach for privacy preserving big data publishing which is based on hadoop (MapReduce). You can also refer this paper for most of privacy preserving data publishing techniques if you are not using any scalable data processing environment such as Hadoop.

http://ssdbm2015.org/day3/S8-3_Zakerzadeh-PrivacyPreservingBigDataPublishing.pdf

http://www.sciencedirect.com/science/article/pii/S0022000014000191

https://www.cs.cornell.edu/~shmat/shmat_nsdi10.pdf

http://ieeexplore.ieee.org/document/6470603/

https://www.cs.sfu.ca/~wangk/pub/FWCY10csur.pdf