FHIR Data Streams: A Quick Approach For Real-Time Processing and Transactional Machine Learning using Apache Kafka

An area that has interested me for many years is the digital evolution of health care systems around the world with real-time data. This area not only has tremendous growth potential, it has the potential to prevent illnesses, deaths, pandemics, and reduce the strain on doctors and staff. We are seeing AI and machine learning being used to detect cancers more quickly and accurately. We are seeing Telemedicine being delivered to remote locations, and streamlining processes to reduce waste in the system. And we are seeing a growing commitment by governments and policymakers to embrace advanced technologies and platforms to help in the digital transformation of health care. At the core of this transformation is health care data, specifically, FHIR data.

What is FHIR? FHIR stands for "Fast Health Interoperability Resources". It is a data specification that is governed by the HL7 international body. FHIR captures health data, from health systems, in a standard format like XML and JSON. The standard covers many health data types (or resources) like: Medication, Patient, Observations, etc. There are about 145 resource types and it is a treasure trove of standard information that can be streamed and analysed by TML. Technologies like HAPI FHIR, created by Smile Digital Health (a pioneer in this area), can be used to implement the HL7 standard on health data for improved interoperability.

Now, FHIR data is great, but data is, more or less, useless if one cannot extract insights for better decision making. This is where TML comes in.

Now, FHIR data is great, but it is, more or less, useless if one cannot extract insights for better decision making. This is where TML comes in.

FHIR data can be intimidating to process. For example each record for a Patient could be over 10K as shown below (this is just 1 record for a patient - there would be millions of messages like this):

{"resourceType":"Observation","id":"e4682c39-b6f7-b76c-65bc-00443a7e4314",
"status":"final","category":[{"coding":[{"system":"https://terminology.hl7.org/CodeSystem/observation-category",
"code":"survey","display":"survey"}]}],"code":{"coding":[{"system":"https://loinc.org","code":"93025-5",
"display":"Protocol for Responding to and Assessing Patients' Assets, Risks, and Experiences [PRAPARE]"}],
"text":"Protocol for Responding to and Assessing Patients' Assets, Risks, and Experiences [PRAPARE]"},
"subject":{"reference":"Patient/2cc604e3-3c17-95e1-c731-58561619ee41"},
"encounter":{"reference":"Encounter/9926e416-9466-5879-190c-e63b566979bc"},
"effectiveDateTime":"2014-06-15T08:30:01-04:00",
"issued":"2014-06-15T08:30:01.148-04:00","component":[{"code":
{"coding":[{"system":"https://loinc.org","code":"76501-6",
"display":"In the past year, have you been afraid of your partner or 
ex-partner?"}],"text":"In the past year, have you been afraid of your 
partner or ex-partner?"},"valueCodeableConcept":{"coding":[{"system":
"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}},
{"code":{"coding":[{"system":"https://loinc.org","code":"93026-3",
"display":"Do you feel physically and emotionally safe where you currently 
live?"}],"text":"Do you feel physically and emotionally safe where you 
currently live?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org",
"code":"LA33-6","display":"Yes"}],"text":"Yes"}},{"code":{"coding":[{"system":
"https://loinc.org","code":"93027-1","display":"Are you a refugee?"}],"text":
"Are you a refugee?"},"valueCodeableConcept":{"coding":[{"system":
"https://loinc.org","code":"LA33-6","display":"Yes"}],"text":"Yes"}},{"code":
{"coding":[{"system":"https://loinc.org","code":"93028-9","display":"In the past 
year, have you spent more than 2 nights in a row in a jail, prison, detention 
center, or juvenile correctional facility?"}],"text":"In the past year, have 
you spent more than 2 nights in a row in a jail, prison, detention center, or 
juvenile correctional facility?"},"valueCodeableConcept":{"coding":[{"system":
"https://loinc.org","code":"LA33-6","display":"Yes"}],"text":"Yes"}},{"code":
{"coding":[{"system":"https://loinc.org","code":"93038-8","display":"Stress is 
when someone feels tense, nervous, anxious or can't sleep at night because 
their mind is troubled. How stressed are you?"}],"text":"Stress is when 
someone feels tense, nervous, anxious or can't sleep at night because their 
mind is troubled. How stressed are you?"},"valueCodeableConcept":{"coding":
[{"system":"https://loinc.org","code":"LA6568-5","display":"Not at all"}],
"text":"Not at all"}},{"code":{"coding":[{"system":"https://loinc.org",
"code":"93029-7","display":"How often do you see or talk to people that you 
care about and feel close to (For example: talking to friends on the phone, 
visiting friends or family, going to church or club meetings)?"}],"text":
"How often do you see or talk to people that you care about and feel close 
to (For example: talking to friends on the phone, visiting friends or family, going to church or club meetings)?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30132-7","display":"5 or more times a week"}],"text":"5 or more times a week"}},{"code":{"coding":[{"system":"https://loinc.org","code":"93030-5","display":"Has lack of transportation kept you from medical appointments, meetings, work, or from getting things needed for daily living?"}],"text":"Has lack of transportation kept you from medical appointments, meetings, work, or from getting things needed for daily living?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}},{"code":{"coding":[{"system":"https://loinc.org","code":"93031-3","display":"In the past year, have you or any family members you live with been unable to get any of the following when it was really needed?"}],"text":"In the past year, have you or any family members you live with been unable to get any of the following when it was really needed?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30122-8","display":"I choose not to answer this question"}],"text":"I choose not to answer this question"}},{"code":{"coding":[{"system":"https://loinc.org","code":"63586-2","display":"During the past year, what was the total combined income for you and the family members you live with? This information will help us determine if you are eligible for any benefits."}],"text":"During the past year, what was the total combined income for you and the family members you live with? This information will help us determine if you are eligible for any benefits."},"valueQuantity":{"value":91045,"unit":"/a","system":"https://unitsofmeasure.org","code":"/a"}},{"code":{"coding":[{"system":"https://loinc.org","code":"76437-3","display":"What is your main insurance?"}],"text":"What is your main insurance?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA6350-8","display":"Private insurance"}],"text":"Private insurance"}},{"code":{"coding":[{"system":"https://loinc.org","code":"67875-5","display":"What is your current work situation?"}],"text":"What is your current work situation?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30138-4","display":"Part-time or temporary work"}],"text":"Part-time or temporary work"}},{"code":{"coding":[{"system":"https://loinc.org","code":"82589-3","display":"What is the highest level of school that you have finished?"}],"text":"What is the highest level of school that you have finished?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30192-1","display":"High school diploma or GED"}],"text":"High school diploma or GED"}},{"code":{"coding":[{"system":"https://loinc.org","code":"56799-0","display":"What address do you live at?"}],"text":"What address do you live at?"},"valueString":"973 Bode Center"},{"code":{"coding":[{"system":"https://loinc.org","code":"93033-9","display":"Are you worried about losing your housing?"}],"text":"Are you worried about losing your housing?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}},{"code":{"coding":[{"system":"https://loinc.org","code":"71802-3","display":"What is your housing situation today?"}],"text":"What is your housing situation today?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30189-7","display":"I have housing"}],"text":"I have housing"}},{"code":{"coding":[{"system":"https://loinc.org","code":"63512-8","display":"How many family members, including yourself, do you currently live with?"}],"text":"How many family members, including yourself, do you currently live with?"},"valueQuantity":{"value":4,"unit":"{#}","system":"https://unitsofmeasure.org","code":"{#}"}},{"code":{"coding":[{"system":"https://loinc.org","code":"54899-0","display":"What language are you most comfortable speaking?"}],"text":"What language are you most comfortable speaking?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA30188-9","display":"Language other than English"}],"text":"Language other than English"}},{"code":{"coding":[{"system":"https://loinc.org","code":"93034-7","display":"Have you been discharged from the armed forces of the United States?"}],"text":"Have you been discharged from the armed forces of the United States?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}},{"code":{"coding":[{"system":"https://loinc.org","code":"93035-4","display":"At any point in the past 2 years, has season or migrant farm work been your or your family's main source of income?"}],"text":"At any point in the past 2 years, has season or migrant farm work been your or your family's main source of income?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}},{"code":{"coding":[{"system":"https://loinc.org","code":"32624-9","display":"Which race(s) are you?"}],"text":"Which race(s) are you?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA4457-3","display":"White"}],"text":"White"}},{"code":{"coding":[{"system":"https://loinc.org","code":"56051-6","display":"Are you Hispanic or Latino?"}],"text":"Are you Hispanic or Latino?"},"valueCodeableConcept":{"coding":[{"system":"https://loinc.org","code":"LA32-8","display":"No"}],"text":"No"}}]}        

The key question becomes how to analyse all these data and extract insights? As mentioned above, FHIR data can be streamed to Apache Kafka. In fact, streaming FHIR data to Kafka is critical to extracting insights with TML quickly.

The key question becomes how to analyse all these data and extract insights? As mentioned above, FHIR data can be streamed to Apache Kafka. In fact, streaming FHIR data to Kafka is critical to extracting insights with TML quickly.

Here is how FHIR data is processed with TML and Kafka in 2 easy steps:

1. Capture FHIR data and stream it to Apache Kafka to a topic say FHIRTOPIC (can be any topic name).
2. Using the MAADSTML python library, directly connect to FHIRTOPIC by using the function viperpreprocesscustomjson. A parameter in this function is called jsoncriteria; this parameter allows us to specify the JSON path in the FHIR data to extract the data we need for analysis.

A parameter in this function is called jsoncriteria; this parameter allows us to specify the JSON path in the FHIR data JSON to extract the data we need for analysis.

How? Well that's easy. Below is an example of jsoncriteria for a specific use case to extract field values from FHIR JSON:

jsoncriteria='uid=code.coding.0.code|code.coding.1.code|componea. nt.0.code.coding.0.code|component.1.code.coding.0.code,filter:resourceType=allrecords,payload=payload.payload~

subtopics=code.coding.0.code,component.0.code.coding.0.code,component.1.code.coding.0.code,medicationCodeableConcept.coding.0.code~\

values=valueQuantity.value,component.0.valueQuantity.value,component.1.valueQuantity.value,medicationCodeableConcept.coding.0.display~\

identifiers=code.coding.0.display,component.0.code.coding.0.display,component.1.code.coding.0.display,medicationCodeableConcept.coding.text~\

datetime=effectiveDateTime~\

msgid=subject.reference~\

latlong=address.0.extension.0.extension.0.valueDecimal:address.0.extension.0.extension.1.valueDecimal'?# add + to join fieldsjsoncriteria='uid=code.coding.0.code|code.coding.1.code|componea. nt.0.code.coding.0.code|component.1.code.coding.0.code,filter:resourceType=allrecords,payload=payload.payload~

subtopics=code.coding.0.code,component.0.code.coding.0.code,component.1.code.coding.0.code,medicationCodeableConcept.coding.0.code~\

values=valueQuantity.value,component.0.valueQuantity.value,component.1.valueQuantity.value,medicationCodeableConcept.coding.0.display~\

identifiers=code.coding.0.display,component.0.code.coding.0.display,component.1.code.coding.0.display,medicationCodeableConcept.coding.text~\

datetime=effectiveDateTime~\

msgid=subject.reference~\

latlong=address.0.extension.0.extension.0.valueDecimal:address.0.extension.0.extension.1.valueDecimal'?# add + to join fields        

a. uid - you need to specify to jsoncriteria the key in the JSON. For example, Patient ID could be the uid. If so, you need to specify the JSON path to the Patient ID value. In the above example, the json path to the value of UID is code.coding.0.display: this means code is the top branch in the json, with coding the next level, 0 indicates that since coding is a json array use the first element, and in this first element the "display" key contains the UID value.

b. filter:resourceType=allrecords - you can filter all FHIR records and process only those that meet the filter criteria for a particular key, in this case we want all resourceType records.

c. subtopics=code.coding.0.code - next you can group uid records by subtopics. This means for each uid (i.e. patient id), the subtopics will only be associated with each patient. In this example, the value of this json path, code.coding.0.code, will be the name of the subtopic.

d. values=valueQuantity.value - this will be the value of the subtopic. Values allows us to process this data for this subtopic for this patient.

e. identifiers=code.coding.0.display - you can also specific any identifier for this value. For example, this could be the location of the patient or their name.

f. datetime=datetime=effectiveDateTime - this specifies the json path of the value of the date time for this message.

g. msgid=subject.reference - this is additional identification of the uid.

h. latlong=address.0.extension.0.extension.0.valueDecimal :address.0.extension.0.extension.1.valueDecimal' - you can even get the latitude and longitude for this patient.

Once you have specified all of the JSON paths, TML (Viper binary) will automatically grab all this information and process it. Specifically, it will gather all of the JSON values, as you are streaming FHIR data to Apache Kafka, then it will process it and apply machine learning to the extracted data, in-memory, and in real-time - NO external databases needed - only Kafka.

Once you have specified all of the JSON paths, TML (Viper binary) will automatically grab all this information and process it. Specifically, it will gather all of the JSON values, as you are streaming FHIR data to Apache Kafka, then it will process it and apply machine learning to the extracted data, in-memory, and in real-time - NO external databases needed - only Kafka.

Once you have processed the data, TML can automatically produce the processed outputs to another Kafka topic in a few milliseconds. Processing FHIR data with Kafka and TML provides an unprecedented way of analysing complex health data with 1 line of code. Using this approach, you can create many FHIR solutions that target specific health use cases like: Disease monitoring or surveillance, Medication Fraud, Patient Data Quality Control, Allergy control and prevention, and a lot more. Each TML solution can be containerized with Docker and deployed with Kubernetes.

Here is a TML solution architecture:

Health organizations and governments have an obligation to provide the best health care to its citizens, and this requires the use of data and advanced real-time technologies that reduce the burden on doctors, care providers and health organizations giving them more time to spend with patients for improved care.

Till next time...