Thanks Susan for your question. I am not sure what exactly you are looking for but I do believe you can build-up a solution that can do this extraction based on web services and xml technology. Using web services you only need to know how to call the service properly and the service does the extraction for you from the data source. Of course to do that you need to check of your data sources support web services technology. Most systems do have web service interfaces for critical data elements. You can check that with the software vendors. If I have more information, perhaps I can assist you.

Susan, you just want utopia! Its the same problem all over the place. Here in England for example we have 43 separate police forces all collecting crime data on more than 2,000 data systems between them! When trying to analyse domestic violence and link up crime incidents, no-crime incidents (999 call out) and A&E attendance for just one area its a nightmare. And then the data controllers start hiding behind data protection laws....

Thanks Amr and Allen for your thoughts. I agree I am looking for some sort of Utopia, but it does seem that so many are looking for the same thing but we still do not have a good solution for us. I am not sure if web services are easy to implement in the healthcare environment if the organisations enterprise systems are not already configured to support this. I have been reading about new generation data federation platforms such as Cirro, Pneuron. Are these platforms more likely to support simplified cross system data interrogation for analysis, assuming that semantics are addressed, and ownership/privacy issues? I guess that in such an ad hoc situation, where there isn't the data cleansing step as in ETL, it is more important to have analytical staff more competent in informatics - to be able to understand and account for the data quality issues that would come through from the source systems?

There are some big challenges here for sure. In the US there are certainly additional layers that may make my answer less applicable for you, but the basic answer to your question in my experience is that the systems makers and many individuals involved in the processes of medical records management do not want integration to be easy. As a result, I found in one project, for example, that the vendor advertised easy integration but actually encrypted much of the key data in an unaccessible blob. Through meetings, it became clear to me they understood that making integration easy would reduce the lock-in they had on their clients. Some tools I have seen really make integration easier. The one I like best is in the population health space. I am attaching a link to it as well as a current project of mine working on answering this question more effectively.

http://www.srgtech.com/news/srg-technology-and-mass-general-hospitals-laboratory-of-computer-science-announce-new-approach-to-population-health-management/

I forgot to mention, I have a friend in Queensland (at the University) who is actually adept at dealing with this type of problem. If you reach out to him I suspect he would talk with you and perhaps figure out how to help.

https://www.business.uq.edu.au/staff/details/andrew-burton-jones

In my opinion, the question title is different from the question description.

“Why is it so difficult to integrate information from disparate clinical systems in hospitals?”

Hospital information integration is difficult because of a) subtle semantic differences between concepts and b) evolving nature of health information. First, there exist semantic details and relationships between data instances that are not captured by current data models and standards. Also, the hospital product is not known in the beginning of the service production process. When a manufacturing company starts to build a mobile phone, the result is defined and fixed. In hospital “product” can change dramatically across the service production process. A preliminary mobile phone was later built to become a sailboat.

These fundamental aspects cause problems when trying to integrate seemingly similar systems and data models in clinical settings. Data and systems are almost the same, they even look technically the same, but actually there exists significant differences that should be made also visible. This leads to problems and complexity in modeling and integration. Medical data needs so much context metadata along with the actual data. Traditionally this context data is implicitly embedded to manual work practices and medical education. Identical data can mean different things if it is created in different context.

“What other technologies exist to assist with combining data across systems more flexibly than a DW.”

I assume that you mean traditional physically constructed centralized data warehouses. At least two trends are emerging: In-Memory and NoSQL. Your examples (Cirro etc) are NoSQL-database applications that are used to extract data from sources and that data is stored physically to another format. In-Memory databases are also used to extract data from the sources but data is not stored physically to disks but kept in memory. I would say that In-Memory databases advertise themselves just like you describe – plug-in, match elements and extract data for your limitless data exploration. NoSQL-databases can probably do same with much larger data sets but are slightly slower in ad hoc-exploration. In practice, they can be also on top of each other. Fragmented source data is extracted and stored to a NoSQL-database and that data pool is then delivered to several In-Memory-data exploration cubes.

However, I think that current generation NoSQL and In-Memory databases have the same or even bigger problems with semantics and audit trail than traditional data warehouses. Their flexibility comes largely from skipping explicit quality control, semantic structures and audit trail capability. They have their own benefits but also some new drawbacks. Just because data elements match, they might not be comparable in semantic level. That’s the problem that data warehouses tried to fix by explicit quality control. In-Memory and NoSQL sacrifice both some technical and organizational quality control levels for other benefits. But the problem – semantics – remains. It’s just not anymore explicitly dealt with but implicit feature. Of course, you can build heavy quality controls and explicit semantic structures on top of NoSQL and In-Memory – but then you would again lose their benefits and end up having similar expensive and inflexible data warehouse with different technology. In the end – use both traditional Data Warehouses and flexible In-Memory databases as long as you know their differences and weaknesses. And you know your data and how and why it was originally created before it was stored to any kind data storage.

The problem by Susan is a socio-technical and systemic problem, and any solution to resolve the integration problem in healthcare must be itself integrated with different perspective approaches based on social, psychological, and technical components. For example, there are many regulations among different units of healthcare that are protecting patients' data to be shared; there are integrity, privacy,and trust issues; there are technical infrastructure and security issues, etc. However, the Cloud Computing from the technical perspective could be a promising technology for sharing information of both healthcare units and citizens self health care management. We are doing research about the topic about how cloud computing can be applied to interoperablity of healthcare system. Maybe we can contribute a small part to the solution.

Hi Susan, thanks for you question. Although I am not that familiar with the healthcare domain, I want to give two further comments on how to support the data exchange and integration across diverse clinical information systems. First, we can harness the benefits of applying interoperability standards, such as HL7, xDT, which may be based on XML, SGML, or other markup languages. Second, we might enhance the searching and information retrieval capabilities by building and using ontologies. However, similar to establishing a data warehouse application, creating an ontology can be costly and time-consuming. Still, I believe that Semantic Web technologies and the idea of Linked Data are key to efficient integration of data from distributed systems.

Thanks for the previous posts to this topic, which have really reinforced some thoughts I have had and added great insight. Especially thanks to Sami for clarifying the difference between the new technologies NoSQL and in-memory systems and traditional data warehouses. The drawbacks and benefits for each are what I was imagining and I guess the difficult part comes in deciding which system is best for which situation - to do with being able to understand the business needs and create the right specifications. But I imagine the costs will also vary - DWs are very expensive to buy and run, but NoSQL sounds like a much 'lighter' setup.

Dominic, thanks for the great resources which I am looking at and also how amazing that you can recommend an expert here in my city! It would be wonderful to make contact with your friend if this area progresses for us. I am particularly interested in his research on the effective use of electronic health records in health authorities, so I hope we can make use of his knowledge in our healthcare service in the future.

Hi Susan, The issue of interoperability is a key challenge in all hospitals all over the world. The simple reality is that unless a hospital has all systems from one vendor, it will have a need to integrate data at some point from different systems.

There are technologies called integration engines in Europe from Bull, IBM, Intersystems, Cloverleaf, etc, but these are expensive and all need a huge amount of configuration and management. Another alternative is to interface systems. And then there is the standards issue - some systems may use HL7, others XML, others EDIFACT for messaging. DICOM is universal in Radiology and imaging, but many other areas have not have compatible standards. For clinical terminology there are probably different (or local) versions of SNOMED, LIONC, etc.

So even if you succeed in integrating data, you may be faced with apples and oranges, or a whole box of fruit - tread carefully!

It is so difficult because the data is naturally complex, and there are usually issues of patient identification in different systems unless a common PAS is used, and even then this is a real issue trying to keep all data in sync. And then there is the safety issue - if you integrate systems you need to make sure you have the right patient with the right data in the right context all the time or it may be dangerous. New safety issues have appeared with the use of integrated IT systems in hospitals, for example there are recent articles in the area of electronic prescribing. Again, tread carefully!

Prof Enrico Coiera from the UNSW has a very good book on all these topics - Guide to Health Informatics (2003), link below.

https://research.unsw.edu.au/people/professor-enrico-coiera/publications

Hope this helps, James Brennan, ex hospital IT manager, Health Informatics Consultant (Ireland)