Pulsar Summit 2022 Report - Aug 19

I am publishing this as a PDF since the first version of this was written natively in LinkedIn and was lost in a weird loading glitch.

I am including a copy of the text here, but the full report is in the PDF.

This will be Issue #44! Previous Issues:

The event was a lot of fun, it was so impressive to see how well it came together. ? The event felt like a giant well-run meetup but with professionalism and gourmet food and surroundings. ? What I mean is that it did not feel corporate at all. ? The atmosphere was open source and community.? ? The hallway talks and discussions between different open source projects and communities were awesome. ? It was amazing to see everyone come together for one event. ? From now on this is what I strive for in every event always.

I follow an RV review series because the host is awesome and I was looking for a travel trailer for a while.? ? Matt’s Reviews follows a formula I want to follow for my review of this awesome event.? ? He does three things that he loved and three things that weren’t perfect.

Three Things I Loved

• The event felt and was driven by, for, and of the community.? It had all the right content, character, and feel that you want in a conference. ? It was fun, full of great content, and a great experience. ? I want events to be like this every time.

• It’s great that things were so well organized and fun, but if there weren’t the amazing speakers giving awesome talks it would not have been great. ? The talks just got better and better. ? We saw amazing new features, future items, great use cases, and amazing demos.? ? I was afraid 30-minute talks were too short, but it worked.???

• The main thing that made it amazing was the people. ? So many great speakers, sponsors, and attendees that every minute was filled with great content, interactions, and fun. ? It was so cool to see so many people I work within the community, new people, and friends from other projects and platforms.

Three Things That Were Not The Best

• Flying to the event. ? Travelling has become much more difficult with so many cancellations and delays.? ? Virtual was so awesome for avoiding this time and money sink. ? The only sinks I like send data to Scylla, Elastic, Delta Lake, Apache Hudi, Clickhouse, and more.
• The event was only one day! ? It was like just a taste, I need more Pulsar.? I need more Scylla, Spring, Flink, NiFi, Clickhouse, Hudi, Delta Lake, Spring, and FLiPN. ? I can’t wait for longer events @ Current and ApacheCon.
• The talks were only 30 minutes long. ? I want so much more, but it does seem to be the sweet spot for getting the crux of the topic. ? Ricardo crushed it with an all-live Pulsar - KoP - KSQLDB demo in docker! ? Wow! ? M1 on Docker is not the best though. ? Fix that Apple! ? Though I am not complaining about my new M1 with 32GB RAM.

Lari (DataStax) did a great talk and has the Reactive framework to add to the Spring - Pulsar module.

https://github.com/lhotari/reactive-pulsar-in-5-minutes?

Michael (Cloudera) came and we got to discuss upcoming Cloudera DataFlow (Apache NiFi) work with Apache Pulsar as well as SQL Stream Builder (Apache Flink) with Pulsar. ? We should have some cool events, articles, demos, and content soon.

https://community.cloudera.com/t5/Community-Articles/Using-Apache-NiFi-with-Apache-Pulsar-for-Streaming/ta-p/337891

https://community.cloudera.com/t5/Cloudera-Stream-Processing/Using-Apache-Pulsar-with-SQL-Stream-Builder/m-p/349917?

Marko and Dominiki (Memgraph) had a great discussion with me about some things that Memgraph can do with Apache Pulsar. ? I see some interesting use cases coming out of this.

https://memgraph.com/docs/gqlalchemy/how-to-guides/streams/manage-pulsar-streams?

Robert (Altinity/Clickhouse) had some great discussions on fast data and is a great source for data knowledge. ? I can’t wait for the Altinity conference on open source data.

https://altinity.com/blog/the-open-source-analytics-conference-2022-is-on-15-november?

Soby Chacko (VMWare / Pivotal) came for discussions on his new experimental Pulsar module for Spring and great discussions were all around. ? I am located near him so we will work on bringing a Spring-Pulsar event to Philly, New Jersey, and/or NYC. ? Stay tuned!

• https://github.com/spring-projects-experimental/spring-pulsar?
• https://spring.io/blog/2022/08/16/introducing-experimental-spring-support-for-apache-pulsar
• https://github.com/spring-projects/spring-integration/issues/3771?
• https://github.com/spring-projects-experimental/spring-pulsar/tree/main/spring-pulsar-sample-apps?
• https://docs.spring.io/spring-pulsar/docs/current-SNAPSHOT/reference/html/

David (StreamNative) was there and he signed his awesome Pulsar book. ? It’s amazing to have a co-worker like David answer the most obscure questions ever.

https://github.com/david-streamlio/table-view-demo?

https://streamnative.io/download/manning-ebook-apache-pulsar-in-action

Decodable sponsored food and were around to chat which is awesome.? They have great articles.

https://www.decodable.co/blog/cleansing-osquery-logs-for-apache-pinot

Schedule of Talks

https://pulsar-summit.org/event/san-francisco-2022/schedule

I missed some great talks by Matteo, David, Lari, Heesung, Kai, and Zach. ? I can’t wait for the videos and slides to come out.

Ecosystem Track (I hosted this one)

Nick (Databricks) did a cool talk on DeltaLake plus Apache Pulsar.

https://pulsar-summit.org/event/san-francisco-2022/sessions/building-reliable-lakehouses-with-apache-pulsar-and-delta-lake

https://github.com/tspannhw/FLiP-Pi-DeltaLake-Thermal?

Addison (StreamNative) and Alexey (OneHouse) did a cool talk on Apache Hudi and Apache Pulsar.

https://pulsar-summit.org/event/san-francisco-2022/sessions/unlocking-the-power-of-lakehouse-architectures-with-apache-pulsar-and-apache-hudi?

https://github.com/tspannhw/pulsar-io-lakehouse

Caito (Ververica) did an awesome talk on Apache Flink, Apache Flink SQL, and using Flink with Apache Pulsar. ? As this is the majority of the FLiP Stack this is my favorite stuff. ? There was a debate on when to use Flink vs Pulsar SQL (Presto/Trino) vs Pulsar PFSQL vs Spark SQL. ? For massively scalable continuous SQL with joins, aggregations, and stateful streaming you should use Flink.

Ricardo (AWS) did a jaw-dropping talk.? ? I did not believe someone would do a talk that was as good as Caito. ? Ricardo’s was an amazing all-demo talk fully documented on Github.? This was amazing to see him walk through such a complex usage of Pulsar’s KoP to run as a full-fledged Kafka cluster allowing for KSQLDB apps, CDC, and Kafka microservices. ? We are planning to do an open source streaming meetup/round table at a Brazilian Churrascaria.

https://github.com/riferrei/is-using-kop-a-good-idea

https://github.com/streamnative/kop

https://talks.riferrei.com/LgUiHy/is-using-kop-kafka-on-pulsar-a-good-idea?

Neng (StreamNative) had the newest and shiniest tech to offer and demo. ? Pulsar Functions utilizing SQL instead of Java, Python, or Golang.

https://pulsar-summit.org/event/san-francisco-2022/sessions/simplify-pulsar-functions-development-with-sql?

Peter (Scylla) is the consummate professional and every talk he does is a masterclass in data. ? Every time he speaks it’s a great learning experience. ? I can’t wait until we do another masterclass or a round table.

https://pulsar-summit.org/event/san-francisco-2022/sessions/distributed-database-design-decisions-to-support-high-performance-event-streaming

For continuous updates on slides, articles, photos, code, and more:

https://github.com/tspannhw/PulsarSummitSF2022

Check out the Twitter stream for #PulsarSummmitSF

https://twitter.com/hashtag/PulsarSummitSF?src=hashtag_click&f=live

Check out Scylla’s awesome Pulsar Summit Roundup

https://twitter.com/i/spaces/1dRKZlaBvwbJB?s=20?

Twitter

https://twitter.com/PulsarSummit?

Website

https://pulsar-summit.org/?

The Pre-Report

https://www.dhirubhai.net/pulse/pulsar-summit-pre-report-2022-tim-spann-/

Unboxing

https://twitter.com/PaaSDev/status/1561438246403440640?s=20&t=2xXb0gOYPQaNbsx-YZTdnA?

UPCOMING

https://github.com/tspannhw/FLiPStackWeekly

Come see me speak at JCONF.DEV in Chicago, Sept 25-28. This is a community-run #Java #JVM #Cloud #BigData conference done right. No sales/product pitches here. Just amazing content. Use this link for a $100 discount on any 2 or 3-day pass cevents.io/JC22100

• September 8, 2022: Comcast Labs Connect PHLAI in Philadelphia
• Sept 13-16, 2022: OSS Summit (Virtual)
• Oct 3-6, 2022: Apache Con, New Orleans, LA.
• Oct 4-5, 2022: Current: Apache Kafka Summit. Austin, TX.
• Oct 25, 2022: KubeCon: Detroit
• Oct 25 - Nov 3: AI DevWorld (Virtual)
• Nov 23, 2022: Big Data EU (Virtual)

I hope to see my streaming friends at Current in Austin.

https://2022.currentevent.io/website/39543/welcome/?

As well as all my streaming and data friends at ApacheCon in New Orleans.

https://apachecon.com/acna2022/schedule.html?

OTHER ITEMS THIS WEEK

• https://www.comcastlabsconnect.com/2022-phlai
• https://www.youtube.com/watch?v=hPZS1ocmWhM
• https://www.singularity-data.com/blog/tutorial-pulsar-risingwave-for-fast-twitter-events-processing
• https://www.youtube.com/watch?v=V6u9NsIS0yY
• https://www.youtube.com/watch?v=4KQ6KTkk2no
• https://www.youtube.com/watch?v=ShjDrvV3MNE
• https://www.youtube.com/watch?v=8xADk1UBd4Q
• https://www.youtube.com/watch?v=QTnYVlmyCOw&t=6s
• https://www.youtube.com/watch?v=eGA5LRoqGJ8
• https://www.youtube.com/watch?v=tRChbhHC5fs
• https://www.youtube.com/watch?v=M3R-jBji0g4
• https://www.youtube.com/watch?v=Vfg091JLp90
• https://www.youtube.com/watch?v=l-fWXRfREB0
• https://www.youtube.com/watch?v=5ydrp0WsKnM
• https://www.youtube.com/watch?v=x5OFvx_Ot5o