You’ve heard of event-driven architectures but what do they actually look like?

Learn what is involved in building an event-driven distributed system and how Go can help you achieve “reliability nirvana”.

Intro

There is a lot of info about how to build resilient services and how to run them on kubernetes but there is one topic that rarely gets much spotlight - event driven architectures.

Utilizing an event driven architecture has massive, amazing benefits - increased reliability, significantly reduced number of dependencies, exposing difficult to reach data to your data science team, simplified development and testing flow and the list goes on and on.

But all of this comes at a significant cost - high complexity that remains daunting even to those who have done it before. You need to figure out how to emit and store events, when to consume them, where to store them and how (and when) to read them back.

In this talk, I will detail what is involved in building a maintainable, reliable and highly performant distributed system that utilizes an event driven architecture. We will walk through my personal experience building such systems and how I utilized Go to achieve “reliability nirvana”.

We will touch on the following topics:

• What is event-driven, why you might want to use it and how you should go about using it
• What makes Go so well-suited for building distributed systems
• Message systems such as Kafka and RabbitMQ
• How protobuf can help you build a solid platform foundation
• What libraries and patterns to use

Outline

In this talk, we will cover the following topics:

• Intro (15 minutes)
  • What problem are we solving?
  • Core concepts
    • What is an event?
    • What is event driven architecture?
    • What is event sourcing?
    • Use cases AKA “Achieving Reliability Nirvana”
    • What makes Go so well-suited for building event-driven software
  • Core components
    • Message envelope (Protobuf, Avro, JSON)
    • Protobuf - not just for gRPC
    • Message production and consumption
    • Message bus tech (Kafka, RabbitMQ, NATS, Redis, AWS SQS, GCP PubSub)
    • Message archiving
    • Message replay
    • Caching
• Architect an event driven distributed system (20 minutes)
  • Diagram the architecture for a system that will be able to receive a 100k requests per second and process them asynchronously and reliably
  • Go over the code structure of an event driven Go application (+ provide template)
  • What libraries should you use?
  • How do you achieve a small dependency footprint?
  • How do you develop software that uses event driven architectures?
• Reality (10 minutes)
  • How long will it take to build something like this and .. should I?
  • I have a monolith… but I want to do this. Where do I start?
  • The real cost of building event-driven systems
  • Common pitfalls

Conclusion

Many folks have heard or read tidbits about event driven architectures but most have not worked with them and do not know the details of what’s involved.

After this talk, an attendee will have learned what is event driven architecture, what benefits it can bring and what is involved in building out such a system in Go.

I have built large, distributed systems that utilize event-driven architectures for several years at companies that deal with high throughput. Most recently, I co-founded a data pipeline company that specializes in extracting data from message busses. The companies’ platform uses an event-driven architecture and is powered by Go.

Given that there is very little information about event driven architectures and even less information about how Go fits in this paradigm, I think it’d be an excellent subject to talk about and shine some light on.

My plan is to utilize our dev environment as a “demo” (running on k8s) to showcase all of the components involved in an event-driven system - producers, consumers, reactors and so forth.

My personal, high level details: I have been writing Go in production for 6+ years, have published several open source libraries and have previously presented on systems, ops and software development topics. I am extremely passionate about Go, love distributed systems and am especially fond of event-driven patterns.