The Publish-Subscribe pattern uses events to break tight coupling between parts of an application. In a microservices architecture, we can use a message broker and integration events to allow microservices to talk to each other indirectly. The different pieces are now coupled with the data contract representing the event (its schema) instead of each other, leading to a potential gain in flexibility. One risk of this type of architecture is breaking events’ consumers by publishing breaking changes in the event’s format without letting them know or without having events versioning in place so they can adapt to the changes. Therefore, it is critical to think about event schema evolutions thoroughly. Most systems evolve, as will events, but since schemas are the glue between systems in a Publish-Subscribe model, it is essential to treat them as such. Some brokers, like Apache Kafka, offer a schema store and other mechanisms to help with these; some don’t.Then, we can leverage the event sourcing pattern to persist those events, allowing new microservices to populate their databases by replaying past events. The event store then becomes the source of truth of those systems. Event sourcing can also become very handy for tracing and auditing purposes since the whole history is persisted. We can also replay messages to recreate the system’s state at any given point in time, making it very powerful for debugging purposes. The storage size requirement for the event store is something to consider before going down the event sourcing path. The event store could grow quite large because we have been keeping all messages since the beginning of time and could grow fast based on the number of events sent. You could compact the history to reduce the data size but lose part of the history. Once again, you must decide based on the requirements and ask yourself the appropriate questions. For example, is it acceptable to lose part of the history? How long should we keep the data? Do we want to keep the original data in cheaper storage if we need it later? Do we even need replaying capabilities? Can we afford to keep all the data forever? What are the data retention policies or regulations the system must follow? Craft your list of questions based on the specific business problem you want to solve. This advice applies to all aspects of software engineering: clearly define the business problem first, then find how to fix it. Such patterns can be compelling but take time to learn and implement. Like message queues, cloud providers offer fully managed brokers as a service. Those can be faster to get started with than building and maintaining your own infrastructure. If building servers is your thing, you can use open-source software to “economically” build your stack or pay for managed instances of such software to save yourself the trouble. The same tips as with message queues apply here; for example, you can leverage a managed service for your production environment and a local version on the developer’s machine.Apache Kafka is one of the most popular event brokers that enables advanced functionalities like event streaming. Kafka has partially and fully managed cloud offerings like Confluent Cloud. Redis Pub/Sub is another open-source project with fully managed cloud offerings. Redis is also a key-value store trendy for distributed caching scenarios. Other offerings are (but are not limited to) Solace PubSub+, RabbitMQ, and ActiveMQ. Once again, I suggest comparing the offerings with your requirements to make the best choice for your scenarios.Now, let’s see how the Publish-Subscribe pattern can help us follow the SOLID principles at cloud-scale:
- S: Helps centralize and divide responsibilities between applications or components without them directly knowing each other, breaking tight coupling.
- O: Allows us to change how publishers and subscribers behave without directly impacting the other microservices (breaking tight coupling between them).
- L: N/A
- I: Each event can be as small as needed, leading to multiple smaller communication interfaces (data contracts).
- D: The microservices depend on events (abstractions) instead of concretions (the other microservices), breaking tight coupling between them and inverting the dependency flow.
As you may have noticed, pub-sub is very similar to message queues. The main difference is the way messages are read and dispatched:
- Queues: Messages are pulled one at a time, consumed by one service, and then disappear.
- Pub-Sub: Messages are read in order and sent to all consumers instead of to only one, like with queues.
I intentionally kept the Observer design pattern out of this book since we rarely need it in .NET. C# offers multicast events, which are well-versed in replacing the Observer pattern (in most cases). If you don’t know the Observer pattern, don’t worry–chances are, you will never need it. Nevertheless, if you already know the Observer pattern, here are the differences between it and the Pub-Sub pattern.
In the Observer pattern, the subject keeps a list of its observers, creating direct knowledge of their existence. Concrete observers also often know about the subject, leading to even more knowledge of other entities and more coupling.
In the Pub-Sub pattern, the publisher is not aware of the subscribers; it is only aware of the message broker. The subscribers are not aware of the publishers either, only of the message broker. The publishers and subscribers are linked only through the data contract of the messages they publish or receive.
We could view the Pub-Sub pattern as the distributed evolution of the Observer pattern or, more precisely, like adding a mediator to the Observer pattern.
Next, we explore some patterns that directly call other microservices by visiting a new kind of Façade: the Gateway.
