We often hear stories from Kafka users about taking nine months to implement production-ready Kafka-based data pipelines. We see customers having 50 teams relying on a single Kafka cluster managed by one person. Data engineers cannot easily simulate a production environment without a complex initial setup. Or data scientists struggle with data integration, building an offline ML pipeline to experiment, reproduce models, and debug them locally. Let's explore how to skip the headache of creating computing clusters, managing partitions, shards, and workers' setup.
The talk demonstrates a different way of implementing a cost-efficient serverless streaming pipeline using pure Python. We will leverage technologies like KEDA (Kubernetes Event-Driven Autoscaling) with integrated NATS Jetstream for the distribution of real-time events and Fission serverless framework to run lambda-like functions on Kubernetes.
๐๐ก๐จ ๐ ๐๐ฆBobur is a developer advocate and speaker specializing in software and data engineering. With over 10- years of experience in IT, he blogs about open-source technologies and the community around them.๐๐ก๐๐ญ ๐ ๐๐จBobur works with companies... Read More →
Today, software teams frequently face significant challenges with Kubernetes operations. Developers often grapple with limited visibility, constrained independence, and restricted access to resources within Kubernetes environments. Concurrently, DevOps teams are tasked with daily support duties such as provisioning environments, monitoring usage, and managing governance across multiple clusters, which prevents them from focusing on core DevOps initiatives. These issues contribute to a poor developer experience, reduced productivity, and increased frustration within teams. This talk will delve into strategies for enabling developer self-service in Kubernetes, emphasizing how to maintain both flexibility and control over infrastructure to enhance overall efficiency and satisfaction.
As the Co-Founder & Chief Product Officer at mogenius, Jan Lepsky is specialized in Platform Engineering, focusing on enhancing DevOps efficiency and empowering developers to work independently with Kubernetes. mogenius provides practical tools that enable developers to define and... Read More →
Sourav Khandelwal, Databricks, Sr Software Engineer
Managing a vast fleet of Kubernetes clusters across multiple cloud providers like AWS, Azure, and GCP is often fraught with inefficient manual processes involving scripts and pipelines. These methods are time-consuming, error-prone, and consume valuable engineering resources. In this session, I will present how we addressed these challenges at Databricks by developing an automated system for the lifecycle management of over a thousand multi-cloud-managed Kubernetes clusters. Our solution features a Kubernetes-style continuous reconciliation mechanism for cluster provisioning and deprovisioning, a fast and reliable cluster state change discovery system integrated with Databricksโ product services, and blue-green cluster rotations (cluster swaps) that allow for seamless upgrades by creating new clusters with updated configurations and shifting workloads without downtime. This automation enables us to implement major infrastructure changes and upgrade Kubernetes versions with low risk through staged rollouts, seamless rollbacks, zero downtime, and minimal operator intervention. I will share our methodologies and experiences in constructing this loosely coupled system that orchestrates product workloads and cloud provider APIs.
I am a seasoned software engineer with over 12 years of experience in designing and managing large-scale platforms in cloud-native environments. At Databricks, I have led and contributed to several innovative projects that have scaled and automated our Kubernetes Compute Platform... Read More →
It's been 10 years since Kubernetes came on the scene and became synonymous with running containers at scale. Suddenly everyone needed to run their apps in Kubernetes.
Or did they?
I'll discuss the different real-world scenarios where Kubernetes makes sense, and where it doesn't. We'll talk about how the hype of technology can get you into trouble, and how to evaluate when Kubernetes is the right call for your business. Don't walk down the dark alley....unless you need to.
Andy Suderman is CTO at Fairwinds, a managed Kubernetes-as-a-Service provider. Andy has worked with cloud native technologies for the last eight years helping organizations adopt and manage Kubernetes. Andy is the creator and primary developer of Goldilocksโan open source tool that... Read More →
Anup Ghatage, Salesforce, Lead Member of Technical Staff Varun Shah, Salesforce, Software Engineer
At Salesforce, our team does Kubernetes deployments at the petabyte scale, with 100+ clusters, 10000+ pods, and 25+ PiB of provisioned storage and innumerable Kubernetes jobs. Operating at this massive scale necessitates an approach that reduces engineering toil. Maintaining such mammoth clusters while keeping security in mind is a whole different challenge.
We specifically talk about how and why we picked immutability and Infra-as-code as the cornerstone of our design and the challenges around it. Immutability provides security and scalability and eliminates config drift. Infra-as-code guarantees that changes to our infrastructure follow the same software lifecycle. but the trade-off is a lack of flexibility to adapt to changing requirements.
In this talk, we go over the unique design choices that help us manage, grow, and configure petabyte-scale deployments at Salesforce.
Anup works on Salesforce's Infrastructure platform.Previously, he has worked on database internals, query processing and storage at SAP, Cisco Systems and other companies for more than 8 years. Anup holds a BS from the University of Pune and an MS from Carnegie Mellon University... Read More →
BE with a specialization in Computer EngineeringMS in Electrical EngineeringWorking on enterprise software since 2009 with companies like Brocade (now Broadcom) / VMware and currently with Salesforce.Using cloud native technologies to scale Salesforce software stack.
We often hear stories from Kafka users about taking nine months to implement production-ready Kafka-based data pipelines. We see customers having 50 teams relying on a single Kafka cluster managed by one person. Data engineers cannot easily simulate a production environment without a complex initial setup. Or data scientists struggle with data integration, building an offline ML pipeline to experiment, reproduce models, and debug them locally. Let's explore how to skip the headache of creating computing clusters, managing partitions, shards, and workers' setup.
The talk demonstrates a different way of implementing a cost-efficient serverless streaming pipeline using pure Python. We will leverage technologies like KEDA (Kubernetes Event-Driven Autoscaling) with integrated NATS Jetstream for the distribution of real-time events and Fission serverless framework to run lambda-like functions on Kubernetes.
๐๐ก๐จ ๐ ๐๐ฆBobur is a developer advocate and speaker specializing in software and data engineering. With over 10- years of experience in IT, he blogs about open-source technologies and the community around them.๐๐ก๐๐ญ ๐ ๐๐จBobur works with companies... Read More →
Today, software teams frequently face significant challenges with Kubernetes operations. Developers often grapple with limited visibility, constrained independence, and restricted access to resources within Kubernetes environments. Concurrently, DevOps teams are tasked with daily support duties such as provisioning environments, monitoring usage, and managing governance across multiple clusters, which prevents them from focusing on core DevOps initiatives. These issues contribute to a poor developer experience, reduced productivity, and increased frustration within teams. This talk will delve into strategies for enabling developer self-service in Kubernetes, emphasizing how to maintain both flexibility and control over infrastructure to enhance overall efficiency and satisfaction.
As the Co-Founder & Chief Product Officer at mogenius, Jan Lepsky is specialized in Platform Engineering, focusing on enhancing DevOps efficiency and empowering developers to work independently with Kubernetes. mogenius provides practical tools that enable developers to define and... Read More →
It's been 10 years since Kubernetes came on the scene and became synonymous with running containers at scale. Suddenly everyone needed to run their apps in Kubernetes.
Or did they?
I'll discuss the different real-world scenarios where Kubernetes makes sense, and where it doesn't. We'll talk about how the hype of technology can get you into trouble, and how to evaluate when Kubernetes is the right call for your business. Don't walk down the dark alley....unless you need to.
Andy Suderman is CTO at Fairwinds, a managed Kubernetes-as-a-Service provider. Andy has worked with cloud native technologies for the last eight years helping organizations adopt and manage Kubernetes. Andy is the creator and primary developer of Goldilocksโan open source tool that... Read More →
