Strategies for effective prompt engineering + other resources

As August winds down, we’re excited to bring you a fresh content selection. This month, our focus shifts to LLM observability fundamentals, an in-depth look at strategies for prompt engineering, a roundup of top Weights & Biases alternatives, ICML 2024 paper summaries, and much more.

Enjoy!

MLOps & LLMOps

> How Cradle Achieved Experiment Tracking and Data Security Goals With Self-Hosted neptune.ai - To start off, we have a story from Cradle ’s team on how Neptune played a key role in helping them satisfy their biggest clients' security standards and reduced complexities involved in file tracking.

> Observability in LLMOps: Different Levels of Scale - Up next, Aurimas Griciūnas talks about the demands for observability when training foundational models, how RAG and agent developers benefit from tracing, and observability challenges in agentic networks.

Guides & tutorials

> LLM Observability: Fundamentals, Practices, and Tools - Moving on, we have an article by Ejiro Onose and Kilian Kluge discussing how standard observability techniques in software and ML evolve by incorporating new tools and practices tailored to the specific requirements of LLM applications.

> LLM Evaluation For Text Summarization - Then, for those of you working with text summarization, in this article, Gourav Singh Bais analyzes different LLM evaluation metrics, highlighting their benefits, limitations, and addressing some open questions in the field.

> Strategies For Effective Prompt Engineering - Coming up next, Lucía Cordero Sánchez explores the nuances of effective prompt engineering. In her article, she discusses the basic as well as more experimental techniques, sharing practical examples and valuable insights gained through her experience.

Tools

> The Best Weights & Biases Alternatives - Following, we have something for those of you working or considering working with Wandb. While it's a well-regarded experiment tracking platform, teams often encounter limitations as they scale their ML/AI efforts. In this article, Kilian Kluge and Abhishek Jha evaluate potential alternatives, addressing crucial factors like pricing models, onboarding, administration, or integration concerns.

> The Real Cost of Self-Hosting MLflow -? Lastly, let’s take a look at MLflow. Although it’s available as open-source software, the hosting costs and responsibilities are substantial and often fall on the shoulders of ML/AI teams. Estimating these expenses can be complex. In this article, Aurimas Griciūnas goes through different deployment strategies to offer a realistic cost estimate.

ICML 2024

This year at [ICML] Int'l Conference on Machine Learning , we dared AI/ML researchers to summarize their papers in under 100 seconds. Not only did most of them succeed, but they did so ahead of time. Here are some of the most notable overviews:

> Summary of the paper “Open LLMs are Necessary for Private Adaptations and Outperform their Closed Alternatives” by Olatunji Iyiola Emmanuel (李白)

> Summary of the paper "LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions" by Victor Agostinelli

This concludes our monthly update! If you found it useful, please pass it along to your contacts.

Cheers!