Spark Use Case Discussion in Italy

Last week in Naples, Italy I was able to engage in a Spark Use case discussion. I gave feedback on potential solutions to updating Hive tables using Spark Streaming from triggered events in a Cassandra database. The concern is that the Hive data events are changing over time and if a new event is loaded in the existing Hive table. The new events would break currently executed processes that are reading from the Hive tables in parallel.

To resolve the use case issue, I discussed the potential solutions below:

1. Using Kafka Topics to broker the events. The existing process can load Hive tables based on the events published to the topic. All new events can still be published to Kafka while the job is executing. Once the job completes than subscribe to Kafka to get new events for the next process run.
2. Use Spark Streaming to load Hive transactional table. Than run a major Hive compaction to rebuild the Hive tables for consumption using the delta data loaded in the Hive buckets.
3. Create a staging Hive table to store new event data and use Hive hql to merge the data into the existing table .

It was a great week working with the client in Naples. Also, I was able to enjoy authentic Neapolitan pizza.