As your organization rapidly grows in scale, so does the number of challenges you face. The growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, number of developers, etc. Coming up with an architecture that can handle all of the data flows in a universal, simple way is key. This talk is about Wix's Kafka-based global data architecture and platform. How we made it very easy for 2,000 Wix microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technology stack they use. All the while offering various tools and features to help adapt to the growing scale and insuring high resilience. // bio Natan Silnitsky is a backend-infra team lead @Wix.com. He leads the Data streaming team in charge of building event-driven libraries and tools on top of Kafka and ZIO. Before that, he was part of a task force that was responsible for building the next-generation CI system at Wix on top of Google's Bazel build tool. Has many years of experience as a developer of large-scale web services - First in .Net, later in Scala. Natan's passions include clean and functional code, dev velocity, and great software design.

You want to introduce ZIO to your existing Scala codebase? Great Idea! It will make your code more efficient, readable, composable, and safe. For the past year, Natan Silnitsky has done this at Wix and has learned a lot about how to do it right. In this talk, Natan will show you how to successfully use ZIO in your legacy service using real-life code examples. You will learn key tips and takeaways including how and when to execute the ZIO runtime; how/when to introduce ZLayers into your codebase; how to make your existing APIs interop with ZIO; and how to have more flexibility on ZManaged resource shutdown. When you're done attending the presentation, you'll be able to introduce ZIO into your existing Scala code base with confidence! About Functional Scala: Functional Scala was founded in 2019 to provide an independent, professional platform for speakers across the entire Scala community, bringing together new and existing Scala developers in an uplifting and welcoming environment that focuses on innovation & excellence. Join the Functional Scala Community on Twitter: https://twitter.com/funscala2021 and https://twitter.com/zivergetech Stay informed on future events: https://www.functionalscala.com/, https://ziverge.com/ Join us in 2021: https://www.eventbrite.co.uk/e/functional-scala-2021-tickets-131620665955 #FunctionalScala2020 #FunScala2020 #FunScala20

The talk from Devoxx Ukraine 2020 ONLINE Fb: www.facebook.com/DevoxxUkraine/ Website: www.devoxx.com.ua During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed system at Wix that can easily handle increasing traffic and storage needs with many different use-cases. In this talk I will share these patterns with you, including: * Consume and Project (data decoupling) * End-to-end Events (Kafka+websockets) * In memory KV stores (consume and query with 0-latency) * Events transactions (Exactly Once Delivery) #devoxxua #java #devoxx

The talk from Devoxx Ukraine 2020 ONLINE Fb: www.facebook.com/DevoxxUkraine/ Website: www.devoxx.com.ua At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar. #devoxxua #java #devoxx

Slides: https://www.slideshare.net/NatanSilnitsky, https://github.com/wix/greyhound Wix has finally released to open-source its Kafka client SDK wrapper called Greyhound. Completely re-written using the Scala functional library ZIO. Greyhound harnesses ZIO's sophisticated async and concurrency features together with its easy composability to provide a superior experience to Kafka's own client SDKs. It offers rich functionality including: - Trivial setup of message processing parallelisation, - Various fault tolerant retry policies (for consumers AND producers), - Easy plug-ability of metrics publishing and context propagation and much more. This talk will also show how Greyhound is used by Wix developers in more than 1000 event-driven microservices.

Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices, mostly written in Scala. We've managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. Our Kafka infrastructure is called Greyhound and was recently completely re-written using ZIO. In this talk you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: How to increase dev velocity of event driven style code. How to optimize working with Kafka in polyglot setting How to support growing amount of traffic and developers. How to tackle multiple DCs environment.

In today's environment of globally distributed large scale web applications, reactive, event-driven design is mandatory. In this talk, Natan showcases how we implemented a reliable, scalable, event-driven, distributed Mega Microservices architecture on top of Kafka but without Kafka streams. We, at Wix Engineering, have built powerful (soon-to-be-os) libraries for data streaming and data storage using Kafka features like compacted logs and exactly once semantics. These libraries offer low latency, high throughput event processing and also high volume data-storage with very simple and easy to understand APIs that work seamlessly together with other technologies such as gRPC. Natan presents the challenges faced while creating and fine-tuning such a system. An architecture that is comprised of a 1,000 micro-services that need to handle incoming traffic from multiple data centers while preserving eventual consistency and high throughput.

Is Exactly Once Delivery a pipe dream? Recent versions of Kafka have claimed they have made it a reality. In this talk I will go over the basic theory of messaging in distributed systems, the different message delivery guarantees and the protocols that implement them. I will focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol between the producer and the consumer. Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall shortcomings. Finally, I will show how it has helped power Wix's event-driven data-streaming and data-storage Infrastructure we hope to open source soon. This infrastructure greatly simplifies building and maintaining services in a highly distributed production environment. It lets developers focus on business logic instead of keep making sure their code is idempotent and fault-tolerant.

This video was recorded at Scala Days Lausanne 2019 Follow us on Twitter @ScalaDays or visit our website for more information http://scaladays.org More information and the abstract can be found here: https://scaladays.org/schedule/building-scala-with-bazel