Meltastic by Meltano: August 2023 Edition

What happened in the Meltano universe in the last four weeks? A ton!

We’re building something new; get in touch!

We’re building a new product and we’d love to get your feedback.

We’d love to chat if you’re building?data-heavy software?and…

• Are struggling with merging data from lots of different databases
• Are fighting multiple API libraries
• Don’t want to worry about data engineering, ETL, or data pipelines at all

Then we might have something for you, and you should give us a call!?

Tap-REST-API-msdk

The?tap-rest-api-msdk?is a kind of Swiss army knife of connectors. And thanks to Steve Clarke, it became really useful! Let’s give you the details: With tap-rest-api-msdk, you can connect to any REST API (most APIs out there).

Now, you can make use of two key features, pagination, and authentication to their full extent! You can do OAuth, use bearer tokens, pass API keys in the header, or use AWS.

For pagination, you can now select from a large list of pagination types and choose one that fits your API.

If you haven’t checked this tap out,?we strongly recommend you do so. It should be your second go-to when a proper tap for your source is not available.

What got shipped?

The team has been heads down this month, bringing improvements and squashing bugs to all things Meltano.

Let's look at some of the more significant items we shipped recently!

?? Meltano Cloud

?? Automated deployments

Now, when you set your deployment to track a git branch, any new commits that appear on that branch will automatically be deployed in Meltano Cloud.

Previously users had to complete an additional manual step to get any new committed changes deployed.

Static deployments are still possible by referencing a particular git hash.

?? Custom Notifications

Cloud customers now have greater flexibility over their notifications.

With the updated config CLI command, you can specify exactly how you want to receive notifications.

Email and webhooks are the initially supported destinations and the payload of the notification can be filtered to exactly how you want it - failures only, successes only, or anything!

meltano cloud config notification set        

? Add new projects from the CLI

New projects can now easily be added from the CLI to your Meltano Cloud account.

This simplifies onboarding for new users as well as our power users that have 10s or 100s of projects they need to onboard.

Meltano Core

v2.20.0 of Meltano Core was released. This was mostly a bug, performance, and doc update release.

MeltanoHub

We added a "Most Popular" listing to the Extractors page.?Check it out!

The Meltano Team and Community have also been busy adding new connectors and utilities to the Hub.

Many of the new connectors are related to our recent blog post about how?LLMs are mostly data pipelines.

Extractors

• Added the MeltanoLabs?variant of tap-jira
• Added the MeltanoLabs variant of tap-mysql. This tap is still under active development. -?Link
• Added the MeltanoLabs variant of tap-beautifulsoup -?Link
• Added the sehnem variant of tap-shopify which supports accessing data via graphql -?Link

Mappers

• Added the map-gpt-embeddings mapper. This can be used to connect to OpenAI to generate embeddings for any data from an extractor -?Link

Loaders

• Added the MeltanoLabs variant of target-pinecone which is a Vector Database for storing embeddings.?Link

Utilities

• Added Tableau utility which can be used to trigger a refresh of Tableu data source -?Link
• Added the dbt Artifacts utility which can be used to process dbt-generated artifacts for other use cases such as model lineage -?Link

SDK

We shipped 2 releases of the SDK:

• v0.31.0
• v0.31.1

with...

• Improved connection handling in SQL targets by sharing a connector instance among stream sinks
• Expose builtin add_record_metadata and batch_config target settings

Meltano Related Content

A lot is happening around Meltano on the web. We’re selecting a few pieces here. Be sure to check them out.

• We published an article on?“LLM Apps Are Mostly Data Pipelines.”
• Matatika published?a new Meltano utility, an article about it,?and?a video!?(Wow)
• The dbt-artifacts utility helps to generate data lineage through Meltano; check it out!

Keep on being?Meltastic!?And if we missed your contribution to the community, just hit us up on?Slack,?LinkedIn, or wherever.?