Event Sourcing

Imagine you're managing a banking application, and you notice in your transaction logs that one user has initiated 2000 transfers in a single session. This raises concerns about the integrity of your transaction processing logic. Is there a flaw in the system causing multiple transactions to be triggered by a single user action? How much money was involved in those transfers? What series of events led to this unusual state of the data? To address such a requirement effectively, implementing Event Sourcing can be extremely helpful.

Basic Terminology

Event Sourcing is a design pattern that meticulously records all changes to the state of an application as events. With Event Sourcing, you gain the capability to reconstruct the entire sequence of events leading to the current state of the data. This ensures transparency and facilitates thorough analysis and resolution of unexpected system behaviors, such as the sudden surge in transactions observed in this case. This article will describe the concepts used for this design pattern.

Every operation that changes data is an "event“ in the context of event sourcing. 

Examples of such events include "transfer initiated", "deposit made" or "withdrawal completed." These events are represented by black post-its in the lower part of the visualization below:

Event Store Database

All of these events are stored in an Event Store Database, and all events referring to a certain object belong to one Stream. You can always replay all the events in a stream to correct potential data corruption. Every update operation is appended to the end of the stream as a new event, ensuring that the entire history of changes is preserved. When an update operation affects multiple streams, the objective is to update all these streams consistently. This is especially challenging during periods of high system load.

In the example of the banking application, essential metrics like account balances, transaction history, and account statuses must be constantly updated to ensure accurate and fair banking operations. 

Speaking in more abstract terms, we have to ensure quick reading operations while keeping the complete history of the data. This can be implemented by using Projections which enable quick access to the current state of the data in the event stream. Each projection represents a view of the data that is more or less displayed directly to the user. The event stream generating the projections still has the complete record of previous events that led to the current state of the data.

In our banking application, when a customer initiates a withdrawal, this action triggers a write operation in the event stream. At the same time, an update operation is executed in the account balance projection table, ensuring that the account balance accurately reflects the transaction. This can be seen in the picture below:

CQRS and Eventual Consistency

We have just touched on another concept that is often linked to event sourcing. 

The pattern of "Command Query Responsibility Segregation" (CQRS) is used to avoid the issue of write and read operations blocking each other. This means that the current data state is not calculated synchronously with every write operation. Instead, every command triggers update requirements on projections read by the user. These update requirements can be processed asynchronously, and the user can still access the current state of the data in time.

This concept of a slight delay between a write operation and the update of the projection is known as Eventual Consistency. Depending on the specific use case, a single write operation might trigger updates in multiple projections.

For instance, when a customer initiates a fund transfer, this action may result in updates to various projections. It could lead to an adjustment in the account balance projection as the funds are deducted, while simultaneously updating the transaction history projection to reflect the transfer. Additionally, the transfer may prompt an update in the fraud detection system and add an entry to the transaction processing queue for further processing. 

This ensures that all relevant aspects of the banking system are updated consistently, albeit with a slight delay. This process can be seen in this diagram:

Conclusion

In our banking application architecture, the events are handled as follows: All modifications to the system are are stored as events in an event store database. 

All events referring to one object are grouped into one stream.Write operations within a stream trigger updates to projections. Data is being processed using the Command Query Responsibility Segregation (CQRS) pattern. This involves separating the handling of write operations (commands) from read operations (queries) which improves performance. By implementing CQRS, we establish a way to independently analyze, optimize and scale each part of the system according to its specific requirements.

We can now easily investigate anomalies like the one from the beginning of this article. Since the even stream is immutable, this is ideal for auditability. In this case, the user who initiated 2000 transactions was identified as a hacker attempting to manipulate banking transactions. We got him in time though, with the help of the concepts described in this article. Event Sourcing FTW!