?? DevOps Weekly #438: Kubernetes v1.31, The Hidden Costs of Scaling Serverless, and Building a Cost-Effective Petabyte-Scale Logging Platform
FAUN - Developer Community
Join thousands of Developers and DevOps enthusiasts. Read the best tutorials, news, tools, jobs for developers ..etc
? Patrons
Achieving end-to-end observability in AWS environments is critical for maintaining your applications' performance, reliability, and security. See how you can achieve a seamless observability experience and discover tips and tricks for optimizing performance, reducing costs, and ensuring your observability practices align with industry best practices.
?? Spread the word and help developers find you by promoting your projects on FAUN. Get in touch for more information.
?? From FAUNers
Organizations that use GitHub Actions for CI/CD pipelines often face limitations with GitHub’s hosted runners, such as resource constraints and security concerns. Deploying self-hosted GitHub Runners on platforms like AWS provides more control over environments and improves efficiency. Configuring self-hosted runners involves setting up EC2 instances, configuring Docker, and integrating with GitHub Actions workflows for improved performance and security in CI/CD processes.
?? Create your FAUN Page if it's not done yet and start sharing your blog posts, news, and tools on FAUN Developer Community, collect badges and more! ?
?? From the web
Dr. Werner Vogels introduced the Frugal Architect mindset at re:Invent 2023, focusing on cost and sustainability as critical requirements alongside security, compliance, and performance. The AWS Well-Architected Framework aligns with this mindset through its six pillars, guiding architects to build secure, reliable, efficient, and cost-optimized systems. Adopting this framework facilitates strategic cost management, enhancing value by aligning spending with key business priorities.
Zomato's production generated over 50 TB of uncompressed logs per day, peaking at 150 million logs per minute. To handle this, they transitioned from Elasticsearch to Clickhouse, leveraging its horizontal scalability and low latency. Custom Golang workers efficiently batched log insertions, using AWS spot instances for cost savings and using a semi-structured schema to optimize data management. They also implemented query throttling mechanisms and advanced monitoring to ensure performance and resiliency.
A new Kubernetes vulnerability affecting the ingress-nginx controller, CVE-2024-7646, allows malicious actors to bypass annotation validation and potentially gain unauthorized access to sensitive cluster resources. The vulnerability has a CVSS v3.1 base score of 8.8 (High), indicating the potential for significant impact on confidentiality, integrity, and availability of affected systems. It is crucial to upgrade to ingress-nginx controller v1.11.2 or later, audit existing Ingress objects for suspicious annotations, limit who can create and modify Ingress objects, enable Kubernetes audit logging, and implement ValidatingAdmissionWebhooks to enforce stricter validation.
In this guest blog post, Asif Sharif, Founder and Chief Technologist of Modern Requirements, describes how DevOps product owners can overcome various challenges by using Copilot4DevOps Plus, an essential AI tool for Azure DevOps. Product owners face time constraints and challenges with requirements management in a DevOps context, leading to stakeholder dissatisfaction, incorrect scope, and higher project costs. Copilot4DevOps Plus offers features designed to help product owners streamline their workflows, improve project outcomes, and reduce errors.
Choco transitioned away from serverless architecture due to escalating costs, latency issues, and limitations on resource usage with Lambda. The company chose Kubernetes to simplify their processes and have more control over their destiny. The migration strategy involved diagnosing the problem, prescribing the solution, and taking coherent action to transition smoothly.
SQL Injection allows attackers to alter queries sent to a database, posing severe security threats. Key prevention strategies include using prepared statements, validating and sanitizing user inputs, employing ORM libraries, and limiting database privileges. Regular security audits and the use of WAFs further help mitigate SQLi risks.
The 11ty Screenshots API, using headless Chromium and Puppeteer, encountered high costs on Vercel's Pro Tier, projecting ~$2000 annually due to 1276 GB-Hrs usage. Migrating to AWS, despite a more complex manual setup, reduced the projected usage to 101 GB-Hrs, making it a cost-effective solution. AWS is suitable for this heavily cached, rarely updated service, as it offers better scalability and lower costs compared to Vercel.
?? News
Dapr v1.14 introduces a new Jobs API for scheduling tasks, enhancing automation and reliability. The alpha Rust-SDK now supports the Jobs API, and a new Scheduler service can manage these jobs on both self-hosted and Kubernetes environments. New features include namespacing for multi-tenancy, dynamic streaming subscriptions, and metrics path matching for better resource management. The transactional outbox feature is now stable, allowing different payloads for pub/sub brokers and databases. The Python SDK now supports the cryptography API.
Kubernetes v1.31 introduces 45 enhancements, with 11 stable, 22 beta, and 12 alpha features. Kubernetes support for AppArmor is now GA, and kube-proxy improved ingress connectivity reliability is stable. Persistent Volume last phase transition time feature is also GA, offering timestamps for phase changes.
Oliver Smith provided an update on recent efforts for Ubuntu 24.10, with the GNOME 47 beta landing for daily builds. Canonical engineers have focused on enhancing the Snap experience, including progress bars in the dock, improved Snapd handling for NVIDIA graphics driver, and seeded Snap tracks for desktop apps. These improvements aim to differentiate Ubuntu and support Canonical's ecosystem.
Visual Studio 2022 v17.11 is now generally available, incorporating community-driven enhancements like improved search, code reviews, and debugging tools. GitHub Copilot offers AI-generated breakpoint suggestions and refined code completions, while new features support Blazor WebAssembly, ARM64, and NPM management. Enhanced profiling tools and a revamped Resource Explorer aim to streamline development workflows.
领英推荐
?? Discussions
GitHub had a major outage with all services down, much worse than Twitter/X's recent stumble. This recurring monthly downtime from GitHub has set a low standard for reliability, showing the service isn't as dependable as it should be. The notion of self-hosting and not relying entirely on GitHub is proving to be wise.
CosmosDB autoscale feature is causing a database to increase request units usage from 400 to 1000 RU/s, despite total request units only at 22.48. Normalized metrics show higher utilization, leading to confusion about database performance.
?? Tools
Realtime log viewer for docker containers.
shippable application packaging
A modern plugin manager for Neovim
Generate Github Actions configurations from Dagger pipelines
Githug is a practical way to learn git, featuring levels that require git commands to complete.
Ngrok FRP Alternative ? Fast ?? Lightweight ? 0 Dependency ?? Pluggable ?? TLS interception ?? DNS-over-HTTPS ?? Poor Man's VPN ?? Reverse &? Forward ?? "Proxy Server" framework ?? "Web Server" framework ?? "PubSub" framework ?? "Work" acceptor & executor framework
?? Spread the word and help developers find and follow your Open Source project by promoting it on FAUN. Get in touch for more information.
?? Did you know?
Did you know that Google.com was once accidentally sold to a former Google employee, Sanmay Ved, for just $12? In an unexpected twist during an exploration of Google's domain sales platform on September 29, Ved was able to purchase the domain. Surprisingly, the transaction went through, and his credit card was charged, granting him ownership for about a minute. Google quickly realized the error and cancelled the transaction. As a nod to their corporate humor, Google initially offered Ved $6,006.13—a numerical pun of "Google"—to buy back the domain. When Ved decided to donate the money to charity, Google doubled the amount to $12,012, which went to the Art of Living India Foundation, supporting education initiatives in India.
?? Meme of the week
??? Quote of the week
“Nothing is built on stone; all is built on sand, but we must build as if the sand were stone. Jorge Luis Borges” ― Titus Winters, Software Engineering at Google: Lessons Learned from Programming Over Time
?? Thanks for reading
?? Never miss an issue
Join FAUN Developer Community and subscribe to our newsletter here .
?? Keep in touch and follow us on social media:
- ??Medium
?? Was this newsletter helpful?
We'd really appreciate it if you could share it with your friends! You can also donate to help us keep this newsletter going.
?? Have a question or feedback?
Feel free to reach out to us at [email protected] . We'd love to hear from you!
?? Want to sponsor our newsletter?
Reach out to us at [email protected] and we'll get back to you as soon as possible.