The following claims are based on a few years of experience developing applications with the functional paradigm. In particular, contrasts of the paradigms have been distilled through porting code samples for Implementing Domain-Driven Design from a Java/C# implementation to an implementation in F#.
The analysis of the object-oriented paradigm herein is based particularly on its manifestation in languages like Java, C# and related languages. (Variations of object-oriented programming without state have characteristics orthogonal to abstract data types (ADTs) employed in functional programming). Functional programming examples are provided in F#.
This topic has been addressed priorly. Peter Norvig discussed how most of the GoF patterns are invisible or simpler in LISP, calling the patterns “programming language bug reports”. Paul Graham echoes the notion. Similarly, Joe Armstrong describes why OO sucks. Although both LISP and Erlang (designed by Armstrong) are dynamically type-checked, the arguments hold just as well for statically typed languages. In fact, the issue of typing is orthogonal.
Arguably, the power of abstraction in functional programming can be a double edged sword. For example, a monoid is in essence a very simple abstraction - an interface consisting of two members. However, without some context, it can be troublesome to sense its value and utility. Abuse of such abstractions can lead to cryptic code, however this isn’t representative of the functional approach.
Finally, the intent herein is not to undermine the object-oriented paradigm. In many ways, well crafted object-oriented code converges upon a functional style. This is no surprise because as stated earlier, the eventual goals of both paradigms are one and the same. It should also be stated that given the present state of the art, functional programming isn’t practical for all domains.
In this post I survey the SOLID patterns followed by tactical patterns of Domain-Driven Design. These patterns are intimately related. The single-responsibility principle fosters the interface-segregation principle; the open/closed principle fosters the Liskov substitution principle. The dependency inversion principle fosters Domain-Driven Design.
The central grievances with object-oriented programming as presented herein are summarized as follows.
The distinguishing characteristics of functional programming that address these grievances are summarized as follows.
The single-responsibility principle states that every class should have a single responsibility where a responsibility is roughly defined as a reason to change. The principle compensates for the anti-pattern where bloated classes play multiple roles. Classes can bloat for a few reasons. A core principle of object-oriented programming is the binding of data structure to behavior. The problem is that optimizing for data structure encapsulation not only weakens composition characteristics, but also hides the underlying problem of explicit state. As a result, object-oriented code typically contains many data structures with relatively few functions per data structure. Adding methods to a class exerts pressure on the single-responsibility principle and reducing the number of methods can either make the data structure difficult to compose or all together fruitless. Furthermore, the simple syntactical cost of declaring a class often compels programmers to marginalize. In less verbose languages, particularly dynamic ones like Ruby and Python, this problem is less prevalent. In my opinion, the importance of this purely mechanical issue must not be understated. A great deal of effort is put into optimizing development with IDEs and other tools, yet optimization can often be achieved at a far more fundamental level.
In functional programming, the fundamental unit of abstraction is the function. Given that a function has a single input and a single output, functions naturally have a single responsibility. One could certainly define arbitrarily generic function, though this would be counterintuitive. Moreover, functions are syntacticly thrifty.
The open/closed principle states that software entities should be open for extension, but closed for modification. The ambiguity of this statement can be resolved through two variations of the principle. The variation due to Bertrand Meyer simply states that existing classes should only be modified in order to correct bugs. This restriction delivers the closed aspect of the principle. The open aspect is delivered through implementation inheritance, or in other words, inheritance with the goal of reuse rather than subtyping. The variation due to Robert C. Martin espouses openness through polymorphism which by definition also provides for closure since extensibility is supported through substitution rather than modification. Unfortunately, substitution often leads to accidental complexity, which must be addressed by yet another principle - the Liskov substitution principle discussed in the following section.
The primary utility of the open/closed principle is confinement of cascading changes while providing for extensibility. This is achieved by designing for extensibility and prohibiting changes to existing entities. Extensibility is attained by fancy tricks with abstract classes and virtual functions. Closure is attained by encapsulation, or rather by the hiding of moving parts. The existence of this principle merely exposes the object-oriented paradigm as a transitional introduction of polymorphism to imperative, state-oriented paradigms.
In a functional language, functions can be substituted at will and as such, there is no need to “design” for extensibility. Functionality requiring parametrization is naturally declared as such. Instead of inventing a concept of a virtual method and inheritance, one can rely on an existing, elementary concept - the higher-order function.
The Liskov substitution principle is essentially a restricted instance of subtyping which aims to guarantee semantic portability across class hierarchies. Portability is achieved by ensuring that whatever is true of a base type is also true of all subtypes. Subclasses must not strengthen preconditions - they must accept all input and initial state that the base class accepts and subclasses must not weaken postconditions - behavioral expectations declared by the base class must be met by the subclass. These characteristics cannot be enforced by the type system alone. The is a relation of inheritance is thus deceptive - hence the need for a compensating principle. As such, the need for this principle demonstrates a pitfall in subtype (inclusion-based) polymorphism. Implicit factoring by class hierarchy imposes needless inclusion restrictions and requires complex principles o place a bound on accidental complexity.
Functional languages favor parametric polymorphism with bounded quantification thereby avoiding some of the pitfalls of inheritance. Informally, functional languages emphasize substitutability and deemphasize implementation reuse since reuse is better achieved through composition. Most ambitions of the Liskov substitution principle are effectively trivial in a functional language.
The interface segregation principle states that no client should be forced to depend on methods it does not use. In essence it is a restatement of the single-responsibility principle for interfaces and reflects the same underlying problem - the difficulty of balancing responsibility assignment, composition and encapsulation in object-oriented design. On the one hand, it is desirable to encapsulate, on the other hand it is desirable to compose. Furthermore, the problem with employing the interface-segregation principle alone is that it doesn’t directly protect against class bloat and in some ways hides the problem.
Functional programming reduces the need for encapsulation by eschewing state and breeds composition at the core. There is no augmented concept of role-based interfaces because function roles are explicit at the onset. Functions are segregated by default.
The dependency inversion principle states that high-level modules should be decoupled from low-level modules through abstractions. In other words, the principle states that code should be structured around the problem domain, and the domain should declare dependencies on required infrastructure as interfaces. Dependencies thus point inward to the domain model. The reason this principle is an inversion is because typical architectures promoted by the object-oriented approach (via layer architecture) exhibit dependency graphs where high-level modules consume low-level modules directly. Initially, this dependency graph seems natural, since in expressing domain models in code one inevitably depends upon the constructs of the language. Procedural programming allows dependencies to be encapsulated by procedures. Subtype polymorphism defers procedure implementation. Unfortunately, use of subtype polymorphism (interfaces) is often overlooked for expressing domain dependencies in object-oriented implementations. Given that infrastructure code is typically more voluminous, the focus of the code drifts away from the domain. Domain-Driven Design was devised in part to balance this drift.
As a matter of course, the declarative and side-effect free nature of functional programming provide for dependency inversion. In object-oriented programming, high-level modules depend on infrastructure modules primarily to invoke side-effects. In functional programming, side-effects are more naturally triggered in response to domain behavior as opposed to being directly invoked by domain behavior. Thus dependencies become not merely inverted, but pushed to outer layers all together.
The Tell, Don’t Ask principle states that you should endeavor to tell objects what you want them to do; do not ask them questions about their state, make a decision, and then tell them what to do. One strategy for this endeavor is the Command/Query Separation principle. The paradox however is that in attempting to strengthen object-oriented code these principles propose a segregation of data structure and behavior. Combining data structure to behavior obfuscates the distinction between observations about an object and behaviors supported the object. In functional programming, observations are data structures and behaviors are functions - the distinction is crystal clear. Furthermore, invariants can be declared with types instead of state making invalid states irrepresentable. For example, the iterator referenced in the statement of the pattern can be succinctly represented as:
When an iterator value is of type End there is no syntacticly valid way to obtain the next element. Not only is the code much shorter, it is more robust.
The concept of the aggregate remains in functional programming, however it isn’t expressed in terms of a class. Instead, it can be expressed as a quintuple, consisting of a set of aggregate states, an initial state, a set of commands, a set of events and a function mapping the set of commands onto the set of events given a state. Cohesion is provided by a module mechanism. The benefit of this formal definition is improved composition and reuse characteristics. A functional F# implementation of a domain-driven design illustrates this approach. There are no dependencies on persistence infrastructure and the same domain model can be used in an event-sourcing implementation, a key-value store as well as an ORM. Moreover, domain event side effects can be delegated to outer layers without reliance on side-effects. Contrast this implementation with an implementation in C# which only supports event sourcing.
Functional languages typically provide immutable record (product) and union (sum) types with auto-implemented structural equality which addresses this pattern trivially. Heavy reliance on state in object-oriented programming makes references or pointers a first class citizen rather than the structure of the data itself. Furthermore, the syntactical cost of declaring value object classes as well as difficulties in operating upon them can lead to primitive obsession. While it is certainly possible to declare immutable classes in object-oriented languages, heavy use of this concept quite simply calls for better tools.
Contrast
with
Domain events are a powerful mechanism for keeping domain models encapsulated. This is accomplished by allowing various handlers from outer layers to register for a domain event. The problem with domain events in object-oriented languages is that the typical implementation is complex and relies on side-effects. Event registrations are typically declared in the composition root and thus it isn’t immediately obvious from the perspective of the publisher which handlers will be invoked. In a functional language, a domain event is simply a value returned by a function in an aggregate. Interested parties can be explicitly registered as filters. This technique is illustrated by the F# DDD example. Returning domain events from aggregate methods in an object-oriented language is prohibitive due to lack of union types and pattern matching.
In imperative object-oriented code, intent leaks through side-effects and through focus on the how rather than the what. Always having to bind behavior to data structure can also be problematic.
If a developer must consider the implementation of a component in order to use it, the
value of encapsulation is lost. If someone other than the original developer must infer
the purpose of an object or operation based on its implementation, that new developer
may infer a purpose that the operation or class fulfills only by chance. If that was not
the intent, the code may work for the moment, but the conceptual basis of the design
will have been corrupted, and the two developers will be working at cross-purposes.
Since functional programming is more declarative, function names and interfaces tend to be more focused on intent rather than the the underlying mechanics. In addition, the interfaces of side-effect-free functions are by nature more revealing because behavior is made explicit through the return value. As a result, in addition to a purely linguistic benefit of naming with intent, intent is also encoded by the type system. This is not to say that expressing intent is effortless in functional languages. Only that it is better supported by the paradigm.
The following excerpt is resounding evidence against the imperative object-oriented programming style. Side-effects are in direct opposition to encapsulation yet all too often they are the most handy tool.
Interactions of multiple rules or compositions of calculations become extremely
difficult to predict. The developer calling an operation must understand its
implementation and the implementation of all its delegations in order to anticipate the
result. The usefulness of any abstraction of interfaces is limited if the developers are
forced to pierce the veil. Without safely predictable abstractions, the developers must
limit the combinatory explosion, placing a low ceiling on the richness of behavior that is
feasible to build.
Unlike imperative programming, functional programming makes side effects an explicitly designated exception - side-effect-free functions are the norm. This pattern is yet another example of how well crafted object-oriented design converges upon a functional style.
Like many patterns rooted in imperative object-oriented design, assertions purport to wield implicit side-effects.
When the side effects of operations are only defined implicitly by their implementation,
designs with a lot of delegation become a tangle of cause and effect. The only way to
understand a program is to trace execution through branching paths. The value of
encapsulation is lost. The necessity of tracing concrete execution defeats abstraction.
As with intention-revealing interfaces, assertions in functional languages are automatically encoded in the return type of a function in addition to the function name. In languages with powerful type systems such as F# and to a greater extent Scala, assertions often can be encoded by types directly making invalid states irrepresentable.
Conceptual contours emerge when domain knowledge permeates the code to a sufficient degree. In object-oriented languages this can be achieved by carefully following principles of Domain-Driven Design.
When elements of a model or design are embedded in a monolithic construct, their
functionality gets duplicated. The external interface doesn’t say everything a client
might care about. Their meaning is hard to understand, because different concepts are
mixed together.On the other hand, breaking down classes and methods can pointlessly complicate the
client, forcing client objects to understand how tiny pieces fit together. Worse, a
concept can be lost completely. Half of a uranium atom is not uranium. And of course, it
isn’t just grain size that counts, but just where the grain runs.
In functional languages, conceptual contours emerge more readily, once again due to the declarative and side-effect free nature of the paradigm. Specifically, clients of the domain model can rely on cohesive functionality attained with composition and yet still have access to constituents without breaking encapsulation.
Closure of operations illustrates yet another example of coercing composition and structure upon object-oriented designs.
Where it fits, define an operation whose return type is the same as the type of its
argument(s). If the implementer has state that is used in the computation, then the
implementer is effectively an argument of the operation, so the argument(s) and return
value should be of the same type as the implementer. Such an operation is closed
under the set of instances of that type. A closed operation provides a high-level
interface without introducing any dependency on other concepts.
Essentially, closure simplifies reasoning about a problem by restricting the domain of discourse. The example of a functional implementation of a domain exhibits this characteristic at a fundamental levels. The operation of applying a domain event is closed under the set of domain states. In terms of persistence, this naturally translates to event-sourcing but also supports persistence in a key-value store or ORM with no required modification.
The overall intent of the aforementioned patterns is to cultivate a declarative design. As witnessed, functional programming is inherently more declarative and therefore more accommodating in this regard. Through declarative design, we can better distill distinguishing characteristics of the domain and reduce or eliminate coupling to orthogonal concerns of infrastructure. Consequently, re-usability, testability, correctness, maintainability and productivity qualities are tremendously enhanced.
]]>The key to understanding the difference between a bounded context and a sub-domain is understanding the difference between a domain and a domain model. The domain, in effect, is the problem to be addressed with a software effort. It is the problem space. A domain can be decomposed into sub-domains which typically reflect some organizational structure. A common example of a sub-domain is Product Catalog or Accounting - the latter possibly a generic sub-domain. A domain model on the other hand is an abstraction of a domain taking whats necessary to satisfy requirements. It has to be created with the cooperation of developers and domain experts. If the design and analysis process was a mathematical function, the model would be its range. In fact, it would have to be a non-injective and non-surjective function:
Essentially, all models are wrong, but some are useful.
While sub-domains delimit the applicability of domains, bounded contexts delimit the applicability of domain models. As such, the bounded context is within the solution space. The relationship between sub-domains and bounded contexts however is deeper still. The goal of a domain driven design is an alignment between the domain and the software. Ideally, there would be full alignment between a sub-domain and a corresponding bounded context. For example, there would be an accounting bounded context that expresses the model of an accounting sub-domain. An alignment between a sub-domain and a bounded context facilitates reasoning within both the solution space and the problem space.
Reality however isn’t always so forgiving. Given that bounded contexts are a software artifact, there is no fundamental force of nature which would firmly bind it to a domain. This is especially true for legacy software developed without a domain-driven approach. This is also true of integrations with 3rd party software which was developed by a different team with a different perspective on a model which is unlikely to be isomorphic to the model at hand. Furthermore, a bounded context can be somewhat technical in nature. For example, a product catalog usually requires a full-text search system which can form a bounded context in its own right. Even though the sub-domain is a product catalog, there are at least two bounded contexts implementing it. Given these constraints, we must accept a degree of elasticity in the association between sub-domains and bounded contexts.
As noted by Paul Rayner at the Lean Coffee Discussion, it is instructive to contemplate the nature of sub-domain boundaries. How are those boundaries determined in the first place? Conway’s Law states:
Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.
According to Conway’s Law, sub-domain boundaries are determined in part by communication structures within an organization. This is often a acceptable demarcation since communication structures have likely been tested and refined over time. From an business perspective, the formation of an organization is driven by a compulsion to reduce transaction costs. From an abstract systems perspective, communication structures strive to reduce coupling by increasing cohesion. These and the coupling requirements among bounded contexts are all shadows of a single principle cast in different directions.
Much like different models can be designed for a domain, a domain can itself be interpreted in different ways. Moreover, there is no canonical interpretation and the interpretation must be continuously evolved based to a great extent on feedback from the domain model. Eric Evans refers to this process as the model exploration whirlpool.
A sub-domain delimits a domain and exists within the problem space. A bounded context delimits the domain model and exists within the solution space. The ideal is full alignment between a sub-domain and a bounded context, however in practice a degree of flexibility must be accepted in this regard. Furthermore, just as the bounded context is guided by a sub-domain, a sub-domain is, in turn ,guided by a bounded context as part of the model exploration whirlpool.
Validation can be a tough topic. In modern object-oriented languages such as C# and Java, a notable reason for the difficulty is the impedance mismatch between exceptions, which are used to handle various types of error conditions, and methods which return normally. The aftermath is a nonuniform interface coupled with reduced autogenesis. In addition to a labyrinthine implementation, usually requiring explicit support from the hosting runtime, exceptions seem to almost invariably lead to bad programming practices. Contrary to their ambition, exceptions compel programmers to defer and ultimately avoid explicit handling of errors. This is because exceptions are highly optimized for the non-degenerate case and despite guidelines and best practices, their fundamental flaws cannot be escaped.
The following is a typical example used to showcase exception deficiencies.
The method interface is not uniform because the result can either be a integer value, which can be captured by assigning the result to a variable, or an exception which can only be captured by a catch block. Furthermore, static verification extends only to the non-exceptional case. In other words, the compiler will issue a warning if one tries to assign the result of this method to a string, but it won’t issue a warning if the result is assigned to an integer variable. This is problematic because the integer type is a proper subset of the method’s effective range - which is a union of the set of integers and an error case.
Unfortunately, the flaws of exceptions extend beyond interface irregularities to friction in designing exception hierarchies as evidenced by the likes of ApplicationException in .NET. Moreover, it isn’t immediately clear whether a given exception should ever be caught. In .NET, it is just as easy to catch an ArgumentException as it is an OutOfMemoryException though it never makes sense to catch the latter since there is not way to handle it. If all of this wasn’t enough, catching exceptions also incurs a performance penalty.
For more about the pitfalls of exceptions take a look at Exceptions are Bad.
The functional programming paradigm addresses the afore-mentioned scenario with explicit typing. Since the range of the Divide function is extended by the possibility of error, the range type is explicitly amplified. One such amplified type in F# is the option type use of which results in the following.
In this example, an erroneous result is explicitly encoded as an empty option. Therefore, the return type of the divide function is Option<int> - an amplified integer. Note that in this case, the interface of the Divide method is uniform because it is captured entirely in the return type. Pattern matching with the match clause allows the compiler to ensure that both result sub-types are handled explicitly. It follows that, unlike in the C# example, it unnatural to write functional F# code which defers handling of exceptional cases. One drawback of this technique however, is that it can become cumbersome to compose functions which return amplified types. We shall address this composition challenge in the following sections.
For a formal look at the origins of functional approaches to addressing exceptions and beyond take a look at Tackling the awkward squad: monadic input/output, concurrency, exceptions, and foreign-language calls in Haskell.
Let us return to the original inventory domain. In the original implementation, validation was performed in the function which executed an inventory item command:
In this example, a failed assertion will raise an exception which will bubble to an outer layer. While it certainly attains the desired result of preventing execution of the behavior, we will refine this implementation with explicit types.
The first order of business is finding an amplified type to encode both successful and erroneous results. While the Option type, used in the integer division example above, captures both cases properly it doesn’t provide any insight into why the result was erroneous. Fortunately, F# provides the Choice type. Much like Option, Choice is a union type, the difference being that it also allows association of an arbitrary value with the second case. We shall use a two-case Choice type to encode success with the first case and an error expressed as a list of string messages with the second case. Thus, the result type of the exec function will be Choice<’TEvent, string list> instead of just ‘TEvent. A basic implementation follows.
In this example, like the successful result, the erroneous result is returned explicitly. While this implementation avoids many of the pitfalls of exceptions, the syntax could certainly use some work. To address the syntax, we shall put the compositional facilities of F# to work.
The validation framework presented here is based largely on the work of Mauricio Scheffer in Validating with applicative functors in F#. This work has been refined and incorporated into fsharpx drawing on powerful, category theory based composition mechanisms contained therein.
Initially, we define a primitive validator builder function.
This is a function which when given a predicate and an error value creates a single parameter function which passes on the parameter if the predicate returns true otherwise returning the error value. This function shall be used to compose more complex validators.
One of the things we would like to do with these validating functions is write code which applies multiple validations on a parameter. If one of the validators fails, we would like to capture the error and continue collecting errors from subsequent validators finally returning a composite error result. If all validators succeed we pass on the parameter. Ultimately, the validation code should compose with code which executes behavior in the non-erroneous case.
Applicative functors will be used to attain the desired degree of composability. A functor is a commonly used structure in Haskell and functional programming in general. It is sort of like a monad but weaker. For the C# programmer, a functor can be thought of as a set of extension method associated with an amplified type. For example, the IEnumerable<T> type, which amplifies T, together with the Select extension method can be regarded as an instance of the functor type class. Intuitively, a functor provides a way to execute a function which operates upon the constituent type instead of the amplified type.
An applicative functor extends the functor type class with additional functions called pure and apply. Pure takes a non-amplified value and creates an instance of the amplified type corresponding to the functor. In the C# IEnumerable case this would entail a function which when given a value returns an IEnumerable containing that value - yield return value;. Apply for the Choice amplified type is defined as follows.
The apply function takes a function wrapped in a Choice type and arbitrary value also wrapped in a choice type. It applies the function, if available, otherwise passing on the contained errors, possibly composing with errors from the arbitrary value. In effect, it handles each of the four cases that a set of two Choice values can be in. In turn, we use these functions to compose the following functions.
The functions lift2, <?>, and |?> in this example form a sort of interface between instances of the Choice type and non-amplified values. The other two functions compose Choice values together passing on the type of either the left or the right value. We can use these functions to refine the syntax in the inventory item example as follows.
In this example from the InventoryItem module, the validName assertion, with the <* operator, composes two validators from the Validator module which ensure that the inventory item name is neither null or empty. In the exec function, the assertion is composed with the returned event using the <?> operator. This operator takes a Choice value on the left and a non-amplified value on the right and composes them into a single Choice value the first case of which is a value of the type of the second operand. The order of the operands can be reversed with the |?> operator which can be read as if and only if. Note the return type of the exec function is Choice<Event, string list>.
In the original example, the Aggregate module defined an aggregate as follows.
In order to incorporate the validation work herein, we change the signature of exec to address erroneous results. The handler must also be changed to handle errors explicitly. The following snippet from the Aggregate module provides an example.
In this example, only a successful executed command results in a commit to the event store. Otherwise, errors are propagated to the caller. The caller can in turn raise an exception or do anything else it may fancy.
Beyond what was showcased in this post are deeper questions about explicit validation. How can we extract validation rules to outer layers such that they can be translated to client side JavaScript? Will a more descriptive error type be required? Can we use F# computation expressions to declare more complex validation workflows?
In this post we discussed the pitfalls of traditional validation techniques involving exceptions. Next, we implemented a validation mechanism which avoids exceptions and which as a result is in better alignment with functional programming. To this end, we were able to draw on basic concepts in category theory which studies composition among mathematical structures. Throughout our approach, the themes of uniformity, explicitness and declarative design prevail. The resulting code remains succinct, draws on static verification and provides better composition facilities. In particular, it provides for equational reasoning which will be a topic of future posts. Additionally, the explicit implementation is simpler both in terms of readability, intuition and the requirements upon the runtime. Finally, the rich compositional facilities of F# allowed a solution that does not short-circuit like exceptions do, allowing clients to obtain all detected errors immediately.
The source for this post is on GitHub.
]]>It appears that many developers, myself included, initially attempt to harness Domain-Driven Design for the object-oriented programming patterns described by Eric Evans in the Domain-Driven Design book. This is no surprise because these patterns, which include concepts such as entities and repositories, are well founded, refined and hardened by years in service. Together, these patterns embody the tactical side of Domain-Driven Design. The other side of DDD consists of strategic principles and methodologies for analyzing and modeling domains as well as systematic techniques for catalyzing the development process.
The most common misconception is that the tactical patterns define DDD or are strictly coupled to DDD. In turn, this often leads to several problems. First, one can be lead to believe that if a design fails to adhere to tactical DDD patterns then it either isn’t DDD or DDD can’t be applied. For example, the anemic domain model anti-pattern is often utilized to demonstrate that a design is not a domain-driven design. This is somewhat misleading because implementation of aggregates, entities and value objects is only a part of the puzzle addressed by Domain-Driven Design. Moreover, the tactical patterns are relevant beyond DDD and can be traced back to established object-oriented design principles. For example, the repository pattern can be regarded as an application of the single-responsibility principle in that it assigns the responsibility of persistence to the repository and encapsulation of behavior to entities. Additionally, the repository facilitates the dependency inversion principle by allowing its implementation to depend on an abstraction declared in the domain layer. Aggregates, entities and value objects exhibit characteristics of the information expert pattern. There is nothing about these principles that is intimately intertwined with DDD - they bring value with or without. Historically, the concept of a domain model, referenced in PoEAA, surfaced before the broad emergence of DDD.
The other problem with fixating on the tactics is ultimately a consequence of the rapid pace of technological change. Principles such as CQRS and event sourcing, while arguably not new, beckon for novel implementation patterns. Designs in functional programming languages can exhibit entirely distinct characteristics as demonstrated in Domain-Driven Design with F# and EventStore. As a result, implementation tactics should remain flexible and open to innovation.
Perhaps more importantly than the tactics, one can overlook the very critical strategic patterns, which can harm the rest of the design regardless of how well applied the tactical patterns are. I’ve experienced this fist hand where a project became increasingly difficult to iterate due to lack of model boundaries. All of the tactical patterns were there - the entities, repositories, aggregates, value objects. Instead of providing value however, these patterns become a burden. It became a burden to ensure that entities contained all the data and behaviors required to fulfill the needs of its consumers. It became a burden to ensure repositories scaled as the number of use-cases increased. It became a burden to partition responsibilities among developers. In the end, the necessary patterns were the bounded context, a context map and a well defined ubiquitous language. The principles of core domain and generic sub-domain, as well as the various relationships between bounded contexts further address demands of complex projects.
The most significant message of the Domain-Driven Design strategy is the predominant importance of domain knowledge. Implementing software is difficult enough, but without distilled domain knowledge, how can we ever hope for it to make any sense? All too often, programmers are blinded by their precious patterns tacitly hoping for the emergence of order. Perhaps this is why Eric Evans, in stating what he’s learned since the book, expressed desire for greater emphasis on the strategic patterns. Luckily for the community, the recently published Implementing Domain-Driven Design by Vaughn Vernon fluently connects the high level strategic patterns to implementation tactics within modern architectures. This book breathes new life into DDD and highlights its ever increasing relevance for today’s software.
]]>Projections implement the query side of CQRS. Specifically in EventStore, projections are a mechanism for transforming event streams into other event streams. This has a wide range of applications including CEP. For this project, projections were used to generate read models, also known as views or query models.
The OverviewReadModelProjection.js projection counts the total number of items in inventory. It does so by selecting events which change the count, namely ItemsCheckedIn and ItemsRemoved. Each occurrence of those events adjusts a state variable and emits a snapshot of the variable as an event. This event can then be retrieved as a read model to serve a query.
As described in the projections series on the EventStore blog, the call to emit emits an event to an event stream with identity “InventoryItemOverviewReadModel”. The fromCategory() function selects events in a specified category. Events are categorized by a built-in projection $by_category which determines the category from the stream ID. The object passed into the when() function contains functions for handling the desired events as well as a state initialization function. The first parameter of the event handling functions is a state variable maintained by EventStore. The state variable can be scoped at the projection level or per event stream, depending on how events are selected. In this case, the state variable is scoped at the projection level. The second variable is the event itself. It has the following structure:
The values contains the stream ID, event body, metadata, event type and other details. Note that this structure is dictated by the serialization format. As such, care must be take to ensure the structure is palatable both in streams, projections and code.
The FlatReadModelProjection.js projection captures the state of individual inventory items including their name, count and active flag.
In this case, events are partitioned by stream with the foreachStream() function. As a result, the state variable will be a associated with each stream. The emitted events will be retrievable using the ID of the source aggregate stream and have category “InventoryItemFlatReadModel”, for example “InventoryItemFlatReadModel-880852396f0f48c6b73d017333cb99ba”.
The projections are retrieved as read models using the ReadStreamEventsBackward function to read the last event:
The read models are declared in F# as follows in the ReadModels module:
Note that the structure of the read models has to match the structure of the events emitted by the projection.
Projections in EventStore are a powerful mechanism with a wide array of applications. In this post, they were shown to support some basic read models. However, some scenarios can call for a document database or full-text search. In such cases events can be dispatched outside of the event store.
The source code for this post can be found on GitHub.
]]>Implementing a design in a programming language once the model is conceived in the mind is often trivial by comparison. As a result, over time, the apparent value of a programming language can become diminished. Regardless of the added features in the next version of C# or Java, they all seem like mere syntactic sugar. F# and the functional paradigm have shifted my perspective on this.
The ambitions of both Domain-Driven Design and functional programming can be regarded as one and the same - that of streamlining the representation of domain knowledge in a formal system with the goal of enabling automation and integration. Functional programming emphasizes declarativeness with use of functions, expressions, immutability and algebraic data structures. These building blocks can be used for both the expression of domain knowledge and the furnishing of a framework for persistent execution of use cases.
The example in this post is based on SimpleCQRS by Greg Young. An existing F# implementation of this project is SimpleCQRS-FSharp by Tuomas Hietanen. A more idiomatic F# implementation is FsSimpleCQRS by Jérémie Chassaing. The example in this post aims to further free the implementation from establishmentarianist object-oriented practices. In fact, the use of classes is contained to integrations with the serialization library and the EventStore API. Classes were avoided in order to explore the benefits and drawbacks of the alternative. Caution was used to prevent focus on technology for the sake of technology alone. As such, the uses of functional techniques are entirely justified.
The core of an OOP DDD implementation consists of classes which represent entities, value objects and domain services. These elements embody the informational core of the project. The remaining components integrate the informational core with infrastructure that connects the domain to databases for persistence, UIs for user interaction and other services.
The informational core in this example is contained in the InventoryItem module:
This single F# source file contains the representation of InventoryItem aggregate state as well as the commands and events associated with the aggregate. Together, these elements capture the entirety of the core domain logic associated with an inventory item. Compared to both the original C# implementation and the referenced F# implementations, this implementation attains a greater degree of encapsulation. By looking at this module alone one could glean an understanding of the functionality of the domain. Furthermore, the use of discriminated unions ensures that all commands and events are explicitly handled with static verification. Unions bring the additional benefit of a uniform interface which is exploited by the Aggregate module described in the next section.
There is no need for an aggregate base class because persistence of state is delegated to infrastructural components defined in outer layers. The identity value is taken out of the domain module because the domain logic itself has no need for it. This allows the identity value type to be defined in outer layers. Once we let go of the aging object-oriented and procedural practices we observe that they are only one of the many options.
The Aggregate module declares an abstraction of an aggregate and uses the abstraction to define a persistent command handler:
The aggregate abstraction is made possible by the use of a uniform interface for executing commands upon aggregate state. Application of persisted events is made with the fold higher-order function - another staple of functional programming. The IntegrationTests module employs the aggregate module to define a command handler by wiring the EventStore and the serialization module.
The EventStore module contains the integration with EventStore. This initial implementation is specific to InventoryItem however it can be easily generalized by generalizing the serializer.
The F# implementation in this post is more concise, more encapsulated, more declarative, more statically verified, and contains far less noise. I would argue that it is also more readable. Explicit factoring by functions elegantly replaces implicit factoring by class hierarchies. This establishes a dependency topology where the informational core is at the center with infrastructural and state related components orbiting in outer layers.
In subsequent iterations, I will expand this example to contain query support via projections and read-models as well as a service layer. A more complex domain will further test this methodology. Specifically, I would like to explore the viability of these techniques with the introduction of behaviors bearing dependencies on domain services and behaviors involving complex invariants. Moreover, I would like to dig deeper into the use of F# types for representing domain state as described in The Designing with types series. The applicability of this approach within scenarios not employing event sourcing is also of interest. F# paves the way for a language-oriented programming paradigm. In particular, computation expressions support the redefinition of control flow constructs in arbitrary contexts. This technique has the potential to define a persistent context for workflows which execute domain logic. This will allow further isolation and encapsulation of the informational domain core.
The source code for this post can be found on GitHub.
]]>One of the distinguishing types in F# as compared to most imperative object-oriented languages is the discriminated union. A discriminated union is an algebraic data type (F-algebra for the mathematically inclined) which consists of a finite number of named cases which themselves can be of any type. A discriminated union value can be of one and only one case. In other words, a discriminated union is a union because the set of cases forms a union. It is discriminated because each case is distinguished from the others by its name. Many F# types including Option and List are defined as discriminated unions.
In C# a discriminated union can be represented as a class hierarchy where the base class corresponds to the union as a whole and sub-classes correspond to the cases. The ability to represent a discriminated union in this way may lead to a dismissive attitude in object-oriented developers. After all, mere brevity of representation, although important, is hardly a reason in and of itself. However, discriminated unions coupled with pattern matching take the discussion to a whole new level.
A central principle of object-oriented programming is the encapsulation of data with related behavior inside boundaries defined by a class. This complexity management mechanism manifest at all levels of a software system, from the smallest objects to entire applications. Objects are also intuitive because of their feigning affiliation with reality. There are however certain deficiencies to the object-oriented approach. The deficiency addressed in this post is based on the observation that classes optimize for extensibility through inheritance thereby hiding inner structure. As a result, whenever behavior needs to depend on sub-classes of a class hierarchy it must either be placed directly into the sub-class or a derived class, or some hierarchy traversal mechanism must be employed.
Class hieararchy traversal can be implemented using several techniques. One such technique is the visitor pattern the goal of which is to augment existing class structures with new behavior without modifying said structures. Consider for instance the ExpressionVisitor from the System.Linq.Expressions namespace. This class is used to traverse the structure of code represented as an AST. Before this class was made public in the .NET Framework a similar implementation was used to implement LINQ support in NHibernate. This visitor pattern facilitates the Open/closed principle by allowing objects, Expression instances in this case, to remain closed because they are not modified or inherited and open in that new behaviors are added - translation to the Hibernate Query Language in this case.
The ExpressionVisitor operates by invoking an internal Accept method on an Expression instance. The Expression instance in turn effectively invokes a visit method on the visitor which corresponds to its specific type. The actual details are slightly more complicated, however the gist of the implementation follows:
To a large extent, the need for the visitor pattern is due to a lack of language support for multiple dispatch. To understand multiple dispatch it is instructive to first consider single dispatch. Single dispatch facilitates polymorphism in object-oriented languages by allowing method invocation to be based on the type of the instance which implements the method. Single dispatch is supported by most object-oriented languages including C# and Java. In multiple dispatch, the method invoked depends not only on the type of the instance, but also on the types of the arguments. The visitor pattern emulates multiple dispatch by employing single dispatch on the Accept method and then resolving the appropriate Visit method either with overloading or calling the method explicitly since the type of the visited object is handy.
The visitor pattern isn’t the only way to implement multiple dispatch. In C# for example, one could invoke the DLR:
This approach, while delegating the dispatch responsibilities to the runtime still suffers from a lack of static verification.
The problem with the visitor pattern is that it is tedious to implement and there is no support from the type system to ensure that all sub-types are accounted for making it error prone. In F#, discriminated unions and pattern matching address both of these problems in an elegant way. The Expression and ExpressionVisitor types could be implemented in the following way:
The Expression is a recursively defined union type which corresponds to the class hierarchy of Expressions. The sample also makes use of the F# tuple type. The print function accepts an expression value and unwraps it using pattern matching. This may initially seem like a glorified switch statement, however it is far more powerful. Most notably, the compiler ensures that all cases are handled. In this way, pattern matching turns any function accepting an expression into a statically checked “visitor”.
Discriminated unions and pattern matching aren’t limited to shaming the visitor pattern. They can also be used to implement the state pattern, also in a statically verifiable fashion. A great example of this can be found in Designing with types: Making state explicit. The resulting design not only prevents invalid state behaviors at runtime, it prevents them at compile time.
We’ve witnessed the elegance with which discriminated unions address OOP deficiencies and lack of multiple dispatch. Discriminated unions and pattern matching allow data structures to be inverted such that they are not only open for extension but extension is guided by support from the type system. Additionally, brevity of expression establishes the discriminated union as an effective modeling tool - after all, a language should be a place to organize one’s thoughts. By contrast, the optimization for extensibility by inheritance in the object-oriented paradigm hides inner object structure making it more difficult to augment class hierarchies. Given the OOP adage of favoring composition over inheritance, this leaves much to be desired in existing object-oriented languages.
]]>The F# option is a type which explicitly represents the presence or absence of a value and is an example of a monad. It is a discriminated union declared as follows.
In C#, Nullable
In JSON, all values can be null, including .NET value types such as Int32. Therefore, an empty option can be represented in JSON with a null. The JsonConverter implementation for the option follows.
The CanConvert method determines whether the converter supports a give type - an option in this case. The typedefof<T> construct returns the generic type definition of the type T. The WriteJson method serializes an instance of the supported type. In .NET, an empty option is a null reference and is serialized as such. Given the earlier observation that options are discriminated unions, the value of a non-empty option can be extracted using the FSharpValue.GetUnionFields helper method. The ReadJson method determines the type wrapped by the option and deserializes the value as usual. Next it creates an instance of an option value using the FSharpValue.MakeUnion helper method.
An F# list represents an immutable series of values of the same type. It is implemented as a linked list and like the option type, it is a discriminated union declared roughly as follows.
This declaration states that a list is either empty or it is a tuple (pair) of a head, which is the first value in the list and a tail which is the remainder of the values in the list. The converter for the list type is defined as follows.
The list is serialized by first being converted to a IEnumerable which Json.NET already supports. Deserializing is achieved by first deserializing into an IEnumerable and then creating a suitable instance of the F# list type. Since a list is a recursive discriminated union, an instance of it is also created recursively with the inner make function which in turn calls FSharpValue.MakeUnion.
An F# tuple is a grouping of unnamed and potentially heterogeneous values. While an analog exists in .NET version 4 and above, a custom converter implementation is still required. JSON does not provide direct support for tuples, but they can be represented in several ways.
One way to represent a tuple in JSON is as an array so that the tuple (“hello”,123) would serialize to [“hello”,123]. This representation has the advantage that arrays already have serialization support and all that is required is the construction of a suitable tuple instance upon deserialization. F# provides the FSharpValue.MakeTuple helper method for creating tuple instances of a specified type given an array of objects denoting the values of the tuple. A first attempt might look like this:
Unfortunately, an object array deserialized in this way may not have the correct type for certain values, numeric values in particular. For instance, the tuple (“hello”, 123) is an instance of the tuple type string * Int32. Upon deserialization however, the second element will have type Int64 not Int32. As a result, each array element must be deserialized individually into a type corresponding to the item in the tuple. This can be done as follows.
The FSharpType.GetTupleElements helper method returns an array of types stored by a tuple. The function readElements in the ReadJson method deserializes array elements individually thus ensuring an appropriate type.
Another way to serialize a tuple is to mirror the serialized shape of a C# tuple so that the tuple (“hello”, 123) becomes {“Item1”:”hello”,”Item2”:123}. This representation is more explicit and will easily deserialize into C# tuples. A excellent account of serializing tuples in this format is depicted in Getting Json.NET to Talk F#, Part 1: Tuples.
F# to C# inter-op isn’t always a thing of beauty. However, the option, list and tuple types are ubiquitous in F# and thus JSON serialization support is essential, especially for enterprise applications. Given these converters, F# types can be used anywhere JSON serialization is required, such as RavenDB which was the motivating factor for these converters. The RavenDB F# client provides generalized support for the union type with a UnionTypeConverter, however the converters in this post result in more idiomatic JSON representations.
The source code for this post can be found on GitHub.
]]>The vast majority of computing today can be decomposed into operations of a Turing Machine. Contrarily, the vast majority of humans think in terms of concepts far beyond symbols on a tape. Perhaps, as alluded to by Douglas Hofstadter in Godel, Escher, Bach, consciousness is merely an illusion established by a balance between self-knowledge and self-ignorance. Self-knowledge is the extent to which we are aware of our thoughts and are able to trace the actions of our mind. Self-ignorance consists of the sub-conscious as well as all of the functions of the central nervous system. Given a thought, we can likely factor it into constituent propositions and statements, which themselves may be further factored. On the other hand, we can’t feel the firing of the underlying neurons or operations of the cerebral cortex.
Natural characteristics of the brain and mind are in turn reflected in the architectures of computing devices. The CPU performs very basic arithmetical and logical operations the fundamental principals of which have remained unchanged since its inception. The software which the CPU ultimately runs however is far more complex than those basic operations. Programming languages and the practice of software engineering have been devised to tame this dichotomy. Yet today, many years after the first CPU and the first program, there remains an ongoing battle between the lower-level forces of hardware and the higher-level forces of domain knowledge.
The battle between man and machine is fittingly illustrated by the contrast between imperative languages and declarative languages. Imperative languages can be thought of as bottom-up abstractions over the underlying hardware. Declarative languages on the other hand are top-down - they represent information and leave it up to the language compiler to translate and convey this information to the underlying hardware. Functional languages in particular are declarative because they are implementations of the lambda calculus on a Turing machine. In terms of practical utility, imperative languages have been winning the battle as evidenced by the predominance of C decades after its creation. The bare-bones simplicity of C and its proximity to the underlying machine are part of the reason for its continual relevance. What this indicates, however, is that programming language technology has yet to attain the level of abstraction and expressive power to make something like C less relevant.
Regardless of the continual prevalence of lower level languages, ambitious attempts at elevating abstraction can provide valuable insight. Take for instance object-oriented programming. Douglas Engelbart envisioned the computer as an extension of the human mind and OOP can be regarded as the incarnation of his vision. Today, OOP is a predominant programming paradigm. The problem is that the promise of object’s capacity to capture the end user’s mental can be deceptive. In the context of GUIs objects serve well in representing the domain. However, for other domains, especially ones based on reality such as LOB applications, OOP’s weaknesses in expressing collaboration can become a notable design and modeling hindrance. OOP can also be somewhat misleading because a class can rarely represent its counterpart in reality to the full extent. For example, a bank account class in an ATM application may model state to represent the available balance and expose behavior for adjusting the balance while protecting invariants. This however represents a small fraction of the functionality required to perform a withdrawal, which also entails aspects such as transactions, server connections, etc. The ATM withdrawal example is drawn from an article on DCI architecture which provides a framework for expressing collaborations between objects based on roles.
The method of action of the DCI architecture facilitates explicit representation of domain knowledge by providing a tailored language of expression as an OOP based framework. DCI was devised in order to compensate for the lack of behavioral expressiveness in traditional OOP. Not surprisingly, similar instances of domain knowledge emphasis abound. An age old mantra in software engineering is the segregation of business logic from presentation and infrastructure logic. This segregation is beneficial not only due to advantages of traditional layering but also due to the emergent isolation of domain knowledge. Alistair Cockburn’s Hexagonal Architecture builds upon this idea and applies it at an architectural level. Domain knowledge is placed at the center with infrastructure components adapting to it. In a sense, knowledge “ripples” from the core throughout components which integrate this knowledge with infrastructure. Another prominent example of knowledge isolation is Domain-Driven Design. A fundamental premise of DDD is placing focus on the core domain, on domain knowledge. The intent is to capture the informational core of the business problem. The remaining components of a working system, while being absolutely essential, are supporting in nature. In retrospect, all of this makes a great deal of sense - after all, computers were designed to solve human problems, not the other way around.
Aspect-oriented programming introduces new mechanisms of composition, partitioning and encapsulation through the notion of a concern. Concerns contain pieces of domain knowledge and the facilities provided by AOP enable composition of concerns and associated behaviors. As a whole, the aspect-oriented paradigm establishes an informational topology wherein knowledge propagates from the core domain out to supporting components. Much like the other paradigms, this type of topology is effective due to its positioning of domain knowledge at the center.
Despite significant advances in programming language theory and software architecture, the I in IT is all too often overshadowed by the T. Reg Braithwaite portrays this phenomenon in Economizing can be penny-wise and pound foolish by coloring code to depict the signal to noise ratio. Green colored code is code that directly express the problem at hand. Yellow colored code represents the accidental complexity of a programming language. Red represents code which has no identifiable function. The goal then is to eliminate red code, reduce yellow code and emphasize the green code.
How can we get there? Where are the weakest links? To some extent the issue is driven by the fact that programming languages carry a double burden. On one hand, a programming language is a place to organize one’s thoughts and express domain knowledge. On the other hand, a programming language must be compiled or interpreted to be ultimately converted into a series of elementary memory manipulation statements. As such, programming languages must be expressive yet simple to use, unambiguous and preferably verifiablee. Expressiveness, simplicity and verifiability are a tough bunch to triage.
Systems such as Hoare logic, algebraic specification languages and denotational semantics are powerful formal verification methods but demand a great deal of sophistication on the part of the programmer and are often impractical as a result. Type systems encompass formal methods which are sufficiently tractable to be widely applicable, yet mainstream programming languages usually support only the tip of the iceberg of the theoretical capabilities. For instance, algebraic data types such as discriminated unions and the associated pattern matching techniques are powerful mechanisms for expressing domain knowledge. Yet these techniques aren’t available in mainstream OOP languages such as Java, C#, etc. They are available in functional languages such as F#, but even most functional languages don’t support higher order techniques such as the polymorphic lambda calculus System F. This seemingly relentless friction leads to some concerning questions. Are modern programming languages approaching the boundaries of the balance between power and accessibility? Will programmers need to embrace more advanced formal techniques in order to advance the state of the art?
All of the above-mentioned paradigms share a common goal of facilitating the conversation between humans and computers. Semantic architectures embody yet another approach to distilling knowledge in software systems. Semantic architectures involve technologies and practices such as ontological engineering, the semantic web and the Web Ontology Language (OWL). These relatively new fields of computer science evolved from the observation that domain knowledge is the the most important aspect of a computer system. In order to be practical, knowledge representation schemes should allow not only for expressive but also for seamless integration with the infrastructure. Ontology languages such as CycL aim to provide such environments. With IT of the Future: Semantic Cloud Architecture Jeff Zhuk outlines a transition from existing SOA architectures to novel knowledge-driven architectures. Knowledge-driven architectures aim to align business and IT and eliminate duplication of knowledge. In this way, they are an evolution of the SOA vision.
]]>Abstraction is frequently associated with the intent to re-use. Identification of an abstraction followed by the preparation of a suitable representation allows code operating upon abstraction to be shared among derived instances of the abstraction. This is traditional OOP polymorphism at play.
Application of abstraction must be judicious because it incurs a non-trivial cost. It forges a dependency chain which in turn requires maintenance the cost of which can outweigh the benefits. Typically, this happens when abstraction focus is misapplied at non-critical caverns far beneath higher level structures in the code. The importance of proper abstraction tends to increase at higher levels of abstraction. A high number of abstractions at low levels results in significant re-factoring friction. Ideally, forces of the DRY principle must be balanced by forces of the YAGNI and KISS principles. A cautionary tale of abstractions is the Limit Your Abstractions series by Ayende.
Encapsulation is a trait of an abstraction. An interface is abstract because implementation is delegated to implementing classes. As a by-product, it also encapsulates the implementation thereby facilitating new semantic levels. New semantic levels however need not be the immediate intent of encapsulation which is also suitable for purely organizational purposes. For example, to improve readability, a private class method can be used encapsulate an operation even if that operation is only invoked in a single place.
These observations can be applied to discussions about the value of certain types of abstractions. There is a debate about the value of the repository abstraction. The repository tends to be a very leaky abstraction because it tends to be difficult to reuse in its entirety across distinct persistence implementations. As a result, significant investment into intricate repository abstraction design ends up as wasted effort - the abstractions are never actually reused. However, the repository abstraction can still reap the benefits of encapsulation. This can be done without any interfaces at all simply by referencing a repository class containing data access methods. This “repository” doesn’t implement an interface and isn’t intended for polymorphism - it is only used to encapsulate.
For example, the Raccoon Blog project avoids repositories and places data access logic directly into the controller. This has the immediate benefit of eliminating two code files - the interface declaration file and the implementation file. On the other hand, it increases the amount of code in the controller. This can make it difficult to distinguish between responsibilities of the controller and responsibilities of the data access layer. Additionally, reasoning about the data access layer of an application becomes trickier because the layer isn’t explicit. Effectively, this is a matter of preference and organizations as well as individual developers can choose an approach best suited for them while considering the implications.
]]>The domain is product pricing of an online retailer and accordingly the primary functionality is the calculation of prices for products based on multiple variables. The central aggregate in this domain is a product pricing strategy which consists of price calculations for each supplier of the product and is keyed by the ID of the product. Prices are updated for a variety of reasons and the trigger can be event based or manual. The events which trigger a price update include any events that may result in a price change, such as a supplier cost change. Manually triggered price updates are invoked by members of the ecommerce team and are happen for a variety of reasons, such as seasonal experimentation. Manual updates are initiated by a selection query which defines the set of products to update. This selection query filters by attributes belonging to products, such as product category, as well as attributes belonging to the product’s pricing strategy, such as margin. Products themselves are aggregates from a different bounded context and can be entities or value objects in the pricing context. Currently, the system manages pricing for approximately 2 million products.
The system runs on the .NET platform, uses SQL Server for persistence and NHibernate for the ORM. Messaging is implemented with NServiceBus. Business logic is implemented using Domain-Driven Design. The functionality is exposed via HTML interface delivered by ASP.NET MVC and ASP.NET WebAPI.
The port project had several goals, including a streamlined mapping with the persistence layer, more advanced querying capabilities and moving as much logic out of the database as possible. A performance improvement was to be a bonus. Additionally, severing the dependence on SQL Server was certainly a plus. Given these constraints and the fact that the existing system was implemented using Domain-Driven Design, a NoSQL document database was deemed an appropriate solution. The document database had to be able to both store and query serialized representations of domain entities. Additionally, some of the more elaborate querying requirements would call for Map/Reduce capabilities.
There are numerous document databases available in the wild that can satisfy the goals of the project. Among the ones considered were RavenDB, MongoDB, CouchDB, SimpleDB, DynamoDB, Azure Table Storage, Cassandra, HBase and Redis. SimpleDB does provide decent querying capabilities, but tables are limited to 10GB and performance seemed questionable. DynamoDB, while highly performant, is a key-value store which means that it doesn’t support secondary indexes. Azure Table Storage has similar drawbacks. Cassandra and HBase can both fulfill all stated requirements, however the configuration and calibration of all moving parts seemed like a hassle. Overall, they are more focused on performance and immense scalability than ease of use. Redis is better suited for storing performance sensitive soft real-time data and it is also a key-value store. Map/Reduce is possible with any NoSQL database, but it requires additional configuration. A good overview of the various NoSQL databases is provided by Kristof Kovacs.
It came down to a decision between RavenDB, MongoDB and CouchDB. All three store serialized representations of entities, support complex queries as well as Map/Reduce. MongoDB has amassed an impressive client list and its indexes are easy to grasp for SQL developers. In CouchDB, all querying is implemented with views which are generated via Map/Reduce. In RavenDB, queries are served by Lucene and the corresponding indexes are defined with LINQ which also supports Map/Reduce declarations. After a bit of exploration, I discovered that a significant advantage that RavenDB had over the others was that its Map/Reduce indexes were updated automatically based on changes in related documents. MongoDB would require an out of band scheduled process to update Map/Reduce collections and CouchDB, by default, only updates Map/Reduce views on demand, not immediately after a document update. While MongoDB indexes are updated in place, some of the more intricate querying requirements would require use of Map/Reduce, as stated. Another advantage RavenDB had over the others is its support of transactions across multiple documents. This is an important feature because many use cases, in addition to storing and updating an entity, store a domain event.
During data migration, MongoDB exhibited better write performance - almost an order of magnitude faster than RavenDB. In testing actual use cases however, RavenDB was on par with MongoDB. Existing use cases involved loading a single aggregate, performing an operation and persisting the resulting changes and domain events. Since migration would only happen once, the advantage MongoDB has in terms of write performance was negligible overall.
Additionally, RavenDB runs on the .NET platform and is open-source. I’ve taken advantage of that on several occasions either by looking at the code to understand certain behavior, or writing extensions to existing functionality. Ultimately, RavenDB was selected as the NoSQL document database for this project.
RavenDB is built on the .NET platform and makes use of two major components, the Esent database engine for document storage and Lucene for indexes. Document storage is ACID while Lucene indexes are updated in the background in an eventually consistent manner. This is effectively CQRS right out of the box! The CAP theorem implies a continuum between ACID and BASE and for this particular project RavenDB resides in a sweet spot between the two. For serialization, RavenDB uses JSON via the Json.NET library. The Unit of Work pattern is supported in a way similar to NHibernate with the IDocumentSession interface which is yet another advantage RavenDB has over the competition. RavenDB supports transactions as a durable resource manager from System.Transactions. Aside from enabling transactional semantics between multiple document sessions, this is helpful for database integration testing by allowing rollback of changes resulting from a test. The NHibernate solution already contained such integration tests which ended up being completely portable. One caveat is that changes aren’t visible until the transaction commits, which is slightly different from the behavior of SQL Server.
RavenDB automatically serializes objects upon persistence and deserializes them upon reconstitution. Serialization is performed automatically but can be customized in several ways. The simplest way is to use the DataMember attribute from the System.Runtime.Serialization namespace. For deeper control, serialization attributes from the Json.NET library can be used.
Beyond that, it is always possible to use an intermediate object which can serve as a DTO between the object model and the database. In this way, the object model is completely decoupled from the persistence layer. However, the intermediate object approach has several drawbacks. One drawback is that it adds a layer of indirection which is prone to error and requires maintenance. Another drawback is that it makes use of the unit of work pattern more difficult, because RavenDB will track changes to the persisted DTO object, not the corresponding domain object.
The best approach to serialization is by convention which can be done by customizing the JsonSerializer and the JsonContractResolver used by RavenDB as described here. The default conventions are mostly satisfactory except that public properties without a setter are also serialized, which of course will throw an exception upon deserialization. It should also be noted that currently, RavenDB creates a new instance of the JsonSerializer when deserializing each individual document. This means that if a customized JsonContractResolver is provided, cache sharing must be enabled so that performance doesn’t suffer.
Another applicable axis of customization is overriding the type names stored for polymorphic properties in the $type field. By default, Json.NET will generate a type discriminator value which is the namespace and assembly qualified name of the type. While this gets the job done, a cleaner, more portable and compact approach is to store a context specific discriminator value. This can be done by overriding the default implementation of SerializationBinder as described here.
Serialization is also important in the context of refactoring. If domain entities are persisted directly then changing the name of a member, or the name or namespace of a type will break serialization. Changes to member names can be facilitated by maintaining a serialization attribute with a hard-coded field name or by using the Patching API to propagate the changes to the entire document set. The former breaks persistence ignorance while the latter can require a lot of processing time. Despite the issues, a document database facilitates a predominant degree of persistence ignorance as compared to their relational database counterparts.
RavenDB indexes are defined through the IndexDefinition class. This class allows the declaration of one or more mapping functions and an optional reduce function with LINQ. It also allows the specification of a post query result transformation function as well as field sorting and indexing options. The map functions select properties from existing documents for indexing and only selected properties will be indexed and made queryable. The optional reduce function can be used to perform aggregation upon properties selected by the map functions providing functionality roughly equivalent to the group by clause.
Indexes can also be defined in a more strongly typed fashion by inheriting from AbstractIndexCreationTask as seen here. In this way, the map, reduce and transform functions are declared directly in the IDE instead of a raw string. For this project however, the strongly typed approach fell short because there was a requirement to index properties of polymorphic types as described here. As a result, I had to resort to the weakly typed IndexDefinition route.
The multi-map/reduce indexing functionality allows selection of properties from documents in multiple collections. This came in handy for this particular project because, as stated, the selection query needs to be able to filter by attributes of both products and pricing strategies. One way to achieve this is by storing the product and pricing strategy in a single document. In this way, the product is a value object of the corresponding pricing strategy, which is not problematic since products are already sourced from a different bounded context. However, given that products have a different life-cycle than the associated pricing strategies and for purely exploratory purposes, a similar outcome was achieved by using a multi-map index. This type of index contains multiple map functions which can select documents from multiple collections, products and pricing strategies in particular. The results are effectively “joined” in the reduce function. The declaration of a multi-map in this type of scenario can be a bit awkward. All of the map functions as well as the reduce function have to produce output having an identical shape. The product and pricing strategy documents contain entirely distinct properties and thus the corresponding map functions had to declare properties that not part of the source document as nulls. Moreover, the reduce function, after joining the documents on a common key which was the product ID, had to select the not-null properties from the group for the final output. Perhaps a better API for this particular scenario would be something similar to the live projections API:
By comparison, the live projections transform function is invoked at query time, not indexing time.
A caveat of RavenDB indexing is indexing speed. Indexes are updated in the background and the system remains responsive during the entire process, but generating the index for the selection query takes several hours and I’ve even experienced it taking over a day on occasion. I found that it is more performant to load all of the data before creating the indexes. This can affect the development, exploration and iteration workflow, because changes to indexes cannot be taken for granted. Generally, this is symptomatic of the shift from relational databases to document databases - no more ad hoc.
RavenDB performs well out of the box but also provides several options for tweaking. For this project like many others, the biggest gain in performance was achieved with batching. As described, one of the use cases involved a selection query which was used to select the products for a price update. The query may select very large sets of products and strategies which would be queued for processing with NServiceBus. NServiceBus allows the batching of several messages into a single transport message. The transport message, which may contain multiple domain messages, is dequeued in a single unit of work and the individual domain messages are dispatched to the appropriate handlers. Andreas Ohlund described a unit of work adapter between NServiceBus and RavenDB which allows RavenDB to batch multiple update requests.
RavenDB includes were used in this project to batch requests to retrieve a product and the corresponding pricing strategy. Includes instruct RavenDB to retrieve a document together with an associated document which will be accessible through the same IDocumentSession. This is a great optimization because it happens behind the scenes and calling code doesn’t have to change.
Another performance gain was achieved by altering the transaction mode with the Raven/TransactionMode setting. Setting the mode to “Lazy” is faster but can result in data loss in case of a server crash. The RavenDB settings documentation also recommends that indexes should be stored on a drive different from the data, though the performance benefit was negligible in this particular project.
All project goals were met, including a sizable performance gain. The mapping between the database and the code changes from a object-relational mapping problem to a serialization problem which is much easier to manage. Furthermore, given Domain-Driven Design, the mapping between aggregate and document is one-to-one which also addresses some of the transactional consistency concerns associated with document databases. The querying capabilities of RavenDB via Lucene are beyond the capabilities of a relational database. Lack of support for anything resembling stored procedures ensures that no business logic resides in the database - only data. Severing the dependence on SQL Server is beneficial for cost, resource and maintenance constraints, which are especially pronounced in a cloud environment. The overall performance gain was three-fold for this particular project, although the testing methods weren’t meant to be precise.
The central “drawback” of document databases in general is the paradigm shift in terms of both data modeling as well as tooling and management. There is bigger pressure to design proper aggregate boundaries and understand querying requirements. This of course isn’t a bad thing.
As a noteworthy piece of trivia, the entire RavenDB client API, including 3rd party libraries, can almost fit on an old-school floppy disk at 1.44 MB as of build 960. For comparison, the NHibernate v3.2 DLL weighs in at 3.9 MB. Together with all required libraries, an NHibernate stack can easily surpass the size of the RavenDB server DLLs which amount to 5.86 MB. Simplicity prevails!
]]>In domain-driven design, one way to attain persistence ignorance is with the use of a repository and an ORM. On the .NET Framework, the ORM NHibernate affords a relatively high degree of persistence ignorance with the use of out of band mapping declarations and reflection. What follows is a summary of various techniques for achieving persistence ignorance with NHibernate.
NHibernate requires that mapped classes provide a default (parameterless) constructor. This allows NHibernate to instantiate the mapped class during reconstitution. Although the footprint is relatively minor, this is an impure and unavoidable constraint for mapped entities. NHibernate could potentially employ a constructor injection technique, as utilized by JSON.NET. If the default constructor is marked as protected then non-reflective client code cannot directly instantiate the class thus still restricting access with other constructors. An ImplicitAttribute could be created to mark protected constructors to prevent productivity tools from flagging them as unused.
NHibernate is capable of mapping database fields to object properties which have private setters. This is one of the most basic methods for maintaining persistence ignorance. Entities and value objects encapsulate domain concepts and as such should regulate access to their internal data. Private setters ensure that class data is only modified by the class itself thereby limiting scope and facilitating encapsulation. With use of access and naming strategies, NHibernate can map to a backing field by convention. This can be useful when having a property setter invoked is undesirable during reconstitution, such as when the setter implements domain logic. If using FluentNHibernate for mapping, private fields having no corresponding property member can be referenced using the Reveal.Property
A common pattern for implementing an aggregation relationship in DDD is with a collection coupled with a method which manipulates it. For example:
The methods AddItem and RemoveItem encapsulate access to the collection of line items in the order entity. The collection can be effortlessly mapped as a one-to-many association. A problem with this approach is that encapsulation can be easily broken because the collection property is publicly exposed and nothing restricts calling code from accessing it directly. To resolve this problem, a backing field access strategy can be used to map the collection to a private field. The property can then return a read-only wrapper around the collection thereby isolating modification of the collection to class methods:
The methods now access the items field directly and attempts to modify the read-only collection returned by the Items property will throw a NotSupportedException.
By default, NHibernate requires class members to be virtual in order to support the proxy pattern; the proxy pattern in turn supports lazy loading. Lazy loading however can be problematic and the requirement to make all members virtual is certainly a infraction against persistence ignorance, despite how negligible. Fittingly, disabling lazy loading at the class mapping level waives this requirement.
To support mapping requirements that cannot be fulfilled by direct field to property mappings NHibernate provides the IUserType and ICompositeUserType interfaces. They enable implementors to declare arbitrary mappings between a set of fields and a component type or value type. Arbitrarily complex value types can be mapped using this technique and since the implementor controls construction the mapped types don’t need to provide a parameterless constructor.
]]>Entities aren’t created in a vacuum nor are they created from a vacuum - normally there are data which are required for creation. For example, consider a simplified purchase order model:
In this case, PurchaseOrder is an aggregate root containing a collection of Invoice instances which themselves contain LineItem instances. In a line of business application, a purchase order would be created to represent a purchase of a set of products or services from a vendor. Subsequently, the vendor sends invoices which are associated with the purchase order. Since PurchaseOrder is an aggregate it manages the act of associating invoices with the purchase order. The association is performed by a method on PurchaseOrder called AssociateInvoice or simply Invoice since invoice can be a verb. A naive implementation could be:
This method certainly gets the job done - it associates an invoice with a purchase order. Unfortunately, this implementation contains flaws and can be improved. For one, because the invoice instance is created outside of the purchase order aggregate, the Invoice.PurchaseOrder property must be publicly settable. This makes it difficult to determine whether a specific invoice instance is associated with a purchase order by looking at code alone. From a business perspective, an invoice is always associated with a purchase order and this constraint should be enforced in the code:
In order to support this implementation, the purchase order has to control the creation of the invoice which means that an Invoice object itself cannot be used as before. Instead, an InvoiceData object can be declared to contain the data representing an invoice. This object is elusively similar to the Invoice object, however it serves a different purpose. It allows the decoupling of the concept of data from the concept of entity thereby supporting the enforcement of domain constraints. Before association with a purchase order invoice data are just data in a vacuum and should therefore be represented as such. This entity data object pattern can be generalized to all cases where entity creation must be strictly controlled.
]]>A primary method for implementing associations is with object references. For example, a customer class may reference an address class which allows calling code to traverse the relationship obtain a customer’s address. Another method for expressing relationships is with a repository. Repositories can express relationships by providing access to associated entities by means of a database search. Eric Evans states:
Whether to provide a traversal or depend on a search becomes a design decision, trading off the decoupling of the search against the cohesiveness of the association. Should the Customer object hold a collection of all the Orders placed? Or should the Orders be found in the database, with a search on the Customer ID field? The right combination of search and association makes the design comprehensible.
Making the decision between employing a reference or a repository is an art of compromise. Furthermore, neither can serve as a substitute for the other in all cases and thus the developer is faced with managing two very different implementation approaches. The important thing to remember is that both methods share a common goal of reflecting the model. A simple way to make the determination is by asking whether a reference is required to support behavior. Often times a reference is used to implement a relationship in order to fulfill a display requirement alone. This can lead one onto a precarious path resulting in the entity becoming increasingly convoluted and intractable. Instead, a repository can be used to implement the association keeping the entity lean. Additionally, the read-model pattern presents a suitable alternative for fulfilling display requirements. The benefits of event sourcing in the context of DDD further promote the notion of keeping entities behavior centric.
Both references and repositories are pertinent implementations of associations in the model. References are inherently cohesive and are thus best suited for associations which are required for behavior or maintenance of integrity. Repositories are best suited for associations which can or must be decoupled from the entity. For example, a customer may be associated with a collection of orders. This collection may become arbitrarily large thus demanding filtering capabilities. Moreover, in a typical scenario, the only behavior in the customer class dependent on the collection of orders is the creating of new orders. The creation of new orders can be just as easily accommodated with a factory and a repository. Repositories can also be utilized instead of lazily loaded references because lazy loading can be problematic.
]]>The pattern consists of the following roles - the client, the service, the message queue, the request processor and the status repository. The client sends the initial request and polls the service for status information. The service receives the request, relays it to a message queue and returns an acknowledgement which typically contains a token to be used for identifying the task in subsequent status queries. The message queue serves to dismantle the temporal coupling between the sending of the request and receipt of the response. The request processor continuously dequeues messages from the queue and performs the actual processing of the request. The status repository is used to store information about the processing status of the task for retrieval by the client via the same service that accepted the initial request or by another service. Various frameworks can be utilized to implement the roles of the request/acknowledge/poll pattern and on the .NET platform a popular combination is ASP.NET WebAPI for the service, NServiceBus for messaging and the request processor and a dead simple implementation of the status repository backed by SQL Server or a plain old file system.
Example Domain
A specific example based on a production system is in order. Consider the domain of an online retailer which sources products from various suppliers, optimizes the content and pricing and posts listings on sales channels. Product information is retrieved from the suppliers website, web service or a flat file. After import, product data is sanitized, optimized and merged with existing data. The import process is seeded by a list of product SKUs provided by the supplier. As one can imagine, importing a product is a long running operation consisting of several steps and thus importing multiple products is also a long running operation. Furthermore, the import of multiple products bears a structure shared by processes where an identical operation is to be repeated for a set of items. The status of such processes typically contains the total number of items to be processed, the number of items processed so far as well as the number of errors.
The service with ASP.NET WebAPI
The service of the request/acknowledge pattern is trivial to implement because it serves a delegating role - a gateway between the client and the messaging infrastructure as well as the status repository. The ASP.NET WebAPI is an abstraction over HTTP which has specifications to fulfill the use case at hand - namely the 202 Accepted response status code as well as the Location header or Link header. Status code 202 indicates to clients that “the request has been accepted for processing, but the processing has not been completed.”. The location or link header can be used to specify the location of the resource representing the status of the process.
In the sample, the import controller implements a controller resource which accepts an import request, generates a unique identifier for the task and creates an internal message to be sent with NServiceBus. Additionally, the service can handle authorization and validation. The model representing the import request, ImportRequestModel, is nearly identical to the internal command message, ImportProducts, however it is best to implement them as separate classes because they have slightly different roles and are handled by different frameworks with different constraints. The task status resource referenced in the location header can also be implemented with the WebAPI.
The queue and request processor with NServiceBus
An asynchronous message queue decouples the sender of a message from the receiver while ensuring message delivery despite downtime of either one. NServiceBus is an abstraction over an underlying messaging technology, typically MSMQ. While there is a managed client for MSMQ, NServiceBus provides several industrial strength facilities beyond MSMQ that will help make the system production ready - namely pub/sub and sagas. For the request/acknowledge pattern, NServiceBus serves as a framework for implementing the queue and request processor roles. The IBus interface encapsulates the queue and a class implementing IHandleMessages will be the request processor. A saga can be used to implement the processing “loop” for importing multiple products. This saga will handle the ImportProducts message and send individual ImportProduct messages to the appropriate endpoint. This endpoint, also implemented with NServiceBus, will handle individual ImportProduct messages and publish an event upon completion or failure. The aforementioned coordinating saga will subscribe to these events in order to manage status information for the import task as a whole. The following is a sample implementation.
This saga implements the request processor with the method for handling the ImportProducts message. Furthermore, the saga entity ImportSagaData is used to track the processing status of the import task. Accordingly, this data is used to update the status of the task with the status repository. In this way, the saga coordinates the import task effectively implementing a distributed, fault tolerant processing loop. The saga data itself can be used instead of the status repository however the status repository is more convenient for a few reasons. First, the saga method MarkAsComplete deletes the saga data entity making the status unaccessible after the task complete. While it is possible to omit the call to the method, leaving it serves as documentation. Second, the saga data is persisted with the ISagaPersister which creates a dependency on the specific implementation. Third, the saga entity isn’t accessible until the initial message completes processing and the entity is persisted. Depending on the size of the task, there may be a number of products that have already been imported at that point without the status being updated.
Utilization of a durable messaging technology, such as MSMQ, makes the system resilient to crashes and downtime all while normalizing the load curve. The endpoint which handles individual ImportProduct messages can go down in the middle of an import and pick up where it left off once it is back up. Furthermore, with the use of the NServiceBus distributor, processing can be scaled out transparently. A caveat in implementing the coordinating saga with NServiceBus v2.6 and below is that the default NHibernate based saga persister isn’t thread safe which means that the endpoint can only have a single worker thread. Given that the work performed by this saga is relatively lightweight this isn’t usually an issue.
The status repository
The status repository can be implemented in a variety of ways and is hardly worth a mention. As a tangential aside, like the ones so forever tantalizing for software engineers, Redis presents an attractive solution for the status repository because of its data structural commands such as INCR. The status repository interface used in the example follows.
Summary
As demonstrated, it is straightforward to implement the request/acknowledge pattern with the ASP.NET WebAPI and NServiceBus. This pattern allows clients to invoke long running operations without having to be bound to the processing time. The service role is implemented with ASP.NET WebAPI. The queue and request processor roles are implemented with an NServiceBus saga, which also coordinates the processing loop. Beyond the elimination of temporal coupling the described system is resilient and horizontally scalable. Despite their conveniences, we aren’t bound to any particular framework or platform - there exist a plethora of messaging technologies and web service frameworks that can fulfill the required roles. For reference, queuing can be supported by MassTransit over RabbitMQ or ActiveMQ. JAX on the Java platform can be used in place of ASP.NET WebAPI. Most importantly, one must understand the role each technology plays and envision the system as being composed with these roles. Care must be taken to ensure that the system isn’t deeply locked into any particular vendor.
Resources
]]>The problem that lazy loading attempts to address can be illustrated with an analogy to the world wide web. The success of the word wide web can be attributed in part to its hyperlinked nature - resources are connected with links allowing for navigation of the web graph loading resources as they are needed. It is unrealistic for the entire web to be loaded into memory, unless you are Google, of course. A relational database can be viewed as a web which ORMs attempt to navigate while at the same time mapping relational data to an object model. The reality is that there is a subtle mismatch between the object model and the relational model. More accurately, No-SQL, corresponding to the object model, and SQL, corresponding to the relational model are mathematical duals as described by Erik Meijer in “A co-Relational Model of Data for Large Shared Data Banks”. (Meijer was also responsible for the Reactive Extensions Framework and demonstrating the duality between IEnumerable
In practice, the mismatch exists because SQL is best suited for ad hoc queries and ad hoc field selection whereas OOP is best suited for static models. From the relational perspective there is a tension to select data specifically for a given query which is in turn designed for a specific use case. From the OOP perspective there is a tension to conform the query to an object model which is designed for a variety of use cases. An important observation is that these mapping issues are most prominent on the query side of the equation. Consequently, a technique such as read-models can be utilized to mitigate lazy loading issues all together. Instead of devising intricate fetching strategies with lazy loading it is much simpler to create read-model classes purposed toward representing queries for specific use cases. In the CQRS and event sourcing world, persistent read-models or projections are a similar technique for implementing queries.
In hindsight, abstraction is absolutely fascinating. Without it, one would be hard-pressed to envision human thought let alone language, mathematics, art, etc. Abstraction breeds generalization and these concepts are perhaps shadows of a single principle cast in different directions. Abstraction itself hinges upon another capacity of thought, symbology, which at the most fundamental level is association between entities. And thus encapsulation is made - with association between a symbol and its meaning. Encapsulation is a mechanism for managing complexity in the face of limits upon resources of the mind. The mind can only have a certain number of things under consideration and without abstraction, generalization and encapsulation, compound ideas could not be formed. We’ve now derived Locke’s three acts of the mind:
The acts of the mind, wherein it exerts its power over simple ideas, are chiefly these three: 1. Combining several simple ideas into one compound one, and thus all complex ideas are made. 2. The second is bringing two ideas, whether simple or complex, together, and setting them by one another so as to take a view of them at once, without uniting them into one, by which it gets all its ideas of relations. 3. The third is separating them from all other ideas that accompany them in their real existence: this is called abstraction, and thus all its general ideas are made.
The concept of abstraction can be illustrated in a variety of ways, all reducible to each other. In a sense, the different portrayals of abstraction are different manifestations and implications of a fundamental axiom. Abstraction can be viewed as representation in that an abstract entity represents a concrete entity. An abstraction can also be defined as a strict subsets of commonalities between things.
In linguistics, abstraction emerges in various forms and one particularly peculiar form is the abstraction hierarchy that is the relation among syntax, semantics, and pragmatics. Syntactics, is the study of relations of signs to one another. This is an abstraction of relations between signs and meanings, which is semantics. Semantics, in turn, is an abstraction of pragmatics which is the study of relations between signs and contextual interpretations. This hierarchical aspect of abstraction is also revealed in linguistics as relations between words ordered by generality, such as for instance the ordering between Noam Chomsky, a specific person and the simply person, a more general term that encapsulates references to any person.
Abstraction is not a human made creation but is present throughout nature. Abstraction at the quantum level allows for the formation of compound structures such as atoms and molecules which in turn combine into cells and ultimately into all forms of life. It is no surprise then that abstraction manifests in human cognition which can be viewed as an extension of the complex chain of abstractions leading up to it, as an emergent behavior. The brain is the most complex structure in the universe and perhaps computer systems will have to emulate biology beyond artificial neural networks in order to exhibit such complexity.
Category theory is a pinnacle of abstraction in mathematics. It purports to formally unify all mathematical disciplines by way of abstractions - collections of objects and arrows. An example from an older discipline of Group Theory, follows. The Abelian group is an abstraction of the familiar integers (more specifically the operation of addition of integers). The group is named after, Niels Henrik Abel, who independently invented group theory to prove a theorem regarding solutions to 5th degree polynomials. This is an example of the power of generality in mathematics - it makes reasoning about certain aspects of mathematics more natural and even elegant.
In 1928, With the Dirac equation, Paul Dirac postulated the existence of the positron. The mathematical model allowed for an electron with positive charge which seemed to contradict experimental results at the time. Ultimately, Dirac’s equation lead to a remarkable discovery in quantum physics and an eerie and intimate relationship between mathematics and reality. This marks yet another mathematical abstraction success story where mathematics, a discipline devised by humans, makes predictions about the physical world. In comparison to other disciplines, mathematics has an advantage in that it is self-defined and purified from real world concerns. Definitions and problem statements are reduced to elementary concepts which allows abstraction to thrive. By contrast, abstractions in programming can become “leaky” due to forces that cannot be removed from consideration.
Abstraction is a cornerstone of computer programming and it echoes throughout the hierarchy starting with machine language, continuing with the C programming language and leading all the way up to user facing application components such as windows, buttons, etc. Low-level programming languages, such as assembly, are termed as such due to the low degree of abstraction between the language and the underlying hardware. Low-level languages are elementary, but difficult to use for construction of complex applications because of the impedance mis-match between human thought and machine codes. Furthermore, a low degree of abstraction implies a high degree of coupling to the underlying hardware thereby reducing portability. High-level programming languages consist of statements and expressions that are closer to natural language. Since high-level programming languages must still run on a computer they are translated to lower level languages, ultimately resulting in machine code. This translation process creates a layer of abstraction, which enables increased portability and expressiveness. The need for translation however is also a cause of trouble for abstractions in computer programming. The reality is that the mechanics of the underlying hardware cannot be entirely escaped and are bound to creep into higher levels if not managed properly. Take for example garbage collection which purports to abstract away memory management. While a powerful abstraction it carries a set of compromises that cannot be overlooked. One is determinism - since required memory manipulation operations are executed behind the scenes, the programmer cannot be certain of the memory state of the system. Another compromise is hidden complexity - while for the most part programmers can forget about memory management, they must also be aware of a complex GC system at play to understand certain intricacies of their programs.
In OOP, classes and interfaces are mechanisms of abstraction. The concept of inheritance in OOP borrows from inheritance in nature and allows specialization and reuse. Classes, however, support only a narrow view of abstraction. Any graduate of an entry level OOP course should be able to understand the difference between a class and an instance of a class - an object. The difference however is very much arbitrary. An instance of a class stores state in fields declared in the class definition. In this way, an instance of a class is distinguished from the class itself - it can store specific values. What if the instance could not only hold state, but can be augmented with new behaviours? Is it still the same “class”, or is it a new thing all together? From the perspective of syntax the difference between a class and an object is evident. From a perspective outside of syntax, an object can be viewed as a class for a whole new set of objects which derive from and specialize the class in some way. Prototype-based programming languages, such as JavaScript, don’t utilize classes as abstraction mechanisms, instead supporting the cloning of objects allowing any object to serve as a prototype for another. This is a more fluid and flexible approach to abstraction, although compromising on some of the benefits of OOP.
Model-Driven architecture aims to raise the level of abstractions in software engineering beyond specific development platforms with use of domain-specific languages and transformation tools. However, as of today, MDA techniques have yet to gain industry acceptance or demonstrate a concrete value proposition. This is due to a variety of reasons and perhaps there is a limit to the degree of abstraction attainable, at least with current approaches.
As another testament to elevating abstractions, John Backus called for the liberation of programming from the Von Neumann style in his Turing Award lecture. The vast majority of modern computers are based on the Von Neumann Architecture and the vast majority of programming languages are abstract isomorphisms of this architecture. This basically means that there is a one-one mapping between the hardware and software. Backus coined the term “Von Neumann bottleneck” in reference to both hardware and software limitations that are byproducts of this isomorphism. In the hardware context there is a literal bottleneck which inhibits performance due to limits to data transfer between memory and the processing unit. In the intellectual sense, there is a bottleneck which inhibits reasoning about programs because the architecture encourages a variable-at-a-time thinking. Backus proposes functional programming languages as an evolution of the Von Neumann style and over 30 years after his lecture, his propositions are becoming mainstream. Unfortunately, we’ve yet to witness significant advancements beyond the Von Neumann bottleneck and it remains one of the challenges of abstraction in computer science.
Although great challenges lie ahead, the fields of software and hardware engineering have had immense success with abstractions. Just consider all the intricacies involved in something as simple as opening a website as elucidated in Dizzying but invisible depth. It is a humbling experience to even begin to fathom the countless moving parts, the years of the evolution of human knowledge, the countless brilliant minds that make this seemingly simple action possible.
Needless abstractions are abstractions that don’t bring value. The definition of value, of course, is subjective which highlights the qualitative nature of abstraction. In other words, not all abstractions are created equal - if abstraction is to create new semantic levels and manage complexity it must be invoked in ways appropriate to a given context. Needless abstractions arise when the programmer’s mental model of the program is reflected directly in code without regard for its utility. An interface may be extracted from a class thereby creating an abstraction, however if the interface isn’t used then it brings no value. Instead it can raise confusion and unnecessary dependencies. Avoiding needless abstractions can be difficult because designing abstractions is what programming is all about. The programmer scans his or hers mental model, detecting commonalities, forming composites, mapping relationships and this process can create abstraction-waste-by-product. An example from enterprise development is the generic repository. While the intent is alluring - to reuse and generalize data access code, the drawbacks outweigh the benefits. Moreover, the apparent benefits are misguided and can be attained using more appropriate methods.
The drive for abstractions is a natural consequence of programming, however it must be kept in balance, as often presented by Ayende with his “Limit your Abstractions” series. Conversely, avoiding abstractions for fear of complexity shouldn’t be the default position, because well designed abstractions have the potential to improve clarity, reuse and ultimately advance the state of the art. Additionally, abstractions can furnish encapsulation, which is much needed in programming where one must be capable of switching between levels of abstraction different by orders of magnitude.
When designing abstractions, it is beneficial to keep in mind principles such as YAGNI and KISS. True wisdom is held in combating complexity with expressiveness, not at the cost of expressiveness - just enough abstraction, but no more. Needless abstractions can emerge at all scopes of software engineering, from the developer to the architect. After all, developers and architects are on different ends of the same spectrum the central distinction being that architects analyze applications at higher levels of abstraction. Many of the driving forces however are the same, which is why many times patterns that apply at the class level are also applicable at the system integration level.
The leap in abstraction between machine language and C, achieved in the early seventies, is yet to be surpassed. For instance, the leap in abstraction between C and C# is minimal and owes more to garbage collection, runtime libraries and syntactic sugar than to semantics and expressiveness. Both languages are third generation. Fourth-generation languages are domain-specific pockets of higher levels of abstraction and declarative-ness, such as SQL. Fifth-generation languages, such as Prolog, are further abstractions away from machine language however we’re still far from working with levels of abstraction accessible to humans. Japan’s failed 5th generation computer project is a paragon of the challenges of abstraction. There seems to be a bottleneck in our current approaches and methodologies. If we encounter challenges with something seemingly simply as object-to-relational mapping then surely we will encounter challenges in mapping program code to natural thought. How do we map Von Neumann machines to a largely uncharted network of billions of neurons? Is there a limit to what can be achieved with current hardware technology? In order to attain greater sophistication, will computers have to become more like biological organisms? Will this compromise determinism?
]]>In domain-driven design, there are two schools of thought regarding validation which revolve around the notion of the always valid entity. Jeffrey Palermo proposes that the always valid entity is a fallacy. He suggests that validation logic should be decoupled from the entity which would defer the determination of the validation rules to invoke until runtime. The other school of thought, supported by Greg Young and others, asserts that entities should be always valid. (I must admit that I side with the always-valid school of though and therefore my statements are be biased.)
The scenarios explored by Palermo are certainly suitable and typical, however solutions involving always-valid entities can be implemented. He contends the following considerations for his example:
Point one addresses the fact that if the user profile entity prevents a null name from being assigned, all application code where a null name might be assigned will blow up. Palmero’s solution is to decouple validation such that code where this might happen invokes a different set of validation rules, if any. An alternative solution is to use a different model designed toward that particular scenario. In fact the read-model pattern can be a fitting approach. The second point discusses error messages in the presentation layer. User error messages certainly are the responsibility of the presentation layer, but the always-valid entity does not imply that error messages in exceptions raised by the entity should be propagated directly to the presentation layer. Contrarily, this is regarded as an anti-pattern and a potential security thread. Instead, the presentation layer, viewed as an adapter in a Hexagonal architecture, should catch and interpret the exception translating it into a form applicable to the UI framework at hand. Finally, the last two points discuss an interesting evolutionary issue. Suppose that a gender attribute is introduced into the user profile entity. It is evident that existing users won’t have a gender specified. This is a realistic business scenario and a from a business perspective the users without a specified gender are simply users without a specified gender. When translated into code, this could mean a gender type of “unspecified”. This gender type can serve as a flag to initiate the workflow which asks the user to specify a gender. For new users, the presentation layer can enforce the rule that a gender must be specified. There is no need to allow the entity to enter an invalid state. From the DDD perspective, validation rules can be viewed as invariants. One of the central responsibilities of an aggregate is enforcement of invariants across state changes.
Jimmy Bogard writes:
If we start looking at command/query separation and closure of operations not only on our service objects but our entities as well, we can treat our entities with a little more respect and not drag them around into areas they don’t really belong. Simply put, if we control the operation side of the equation, why in the world would we allow our entities to get into an invalid state? Life becomes much more complicated if we start having “IsValid” properties on our entities.
These succinct statements carry a great deal of information. First is the idea of the “IsValid” property. A requirement to invoke validation or to query an “IsValid” property requires calling code to be non-atomic and this can lead to inconsistencies and a greater potential for human error. It is a difference between:
and this:
The second code sample requires clients of the UserProfile class to be aware of the “IsValid” property and always use it consistently. The first code sample avoids this all together - the operation of instantiating a user profile is atomic. This is a good example of leveraging programming language constructs to represent real world constraints. The next important part of Bogard’s statement is “not drag them around into areas they don’t really belong” which leads into the subsequent section on application layers. If entity validation seems inappropriate in a certain area then this may be an indication that an entity doesn’t belong in that area.
All sufficiently complex enterprise applications consist of multiple layers. From a user’s perspective the layers are abstracted away and they exist solely to assist the programmer in managing all of the emergent complexity. Distinct layers imply that translation must happen between the layers in order for information to propagate. For example, in a typical enterprise use case, an entity is loaded from the database, operated upon, persisted back to the database and information regarding the operation is returned to the user through a presentation layer via perhaps a REST adapter. Applications layers imply the existence of boundaries and as per Mark Seemann’s post, At the Boundaries, Applications are Not Object-Oriented. The entity is contained within the domain layer and should not be dragged into areas it doesn’t belong. In the presentation layer, a specific MVC view may require a user to enter a name and then gender. After having entered a name, the gender is still unspecified and the target entity is an invalid state. An always-valid entity cannot be bound to this view and it fact it should not be bound to the view - this is what the view model is for. The view model is a building block of the presentation layer and the domain entity doesn’t belong there. Instead, an appropriate domain layer entity should be created based on data contained in the view model. This can be done directly or by passing a DTO to a service.
Validation can be implemented with trivial if-then control flows but this can become cumbersome and the programmer’s answer is the validation framework. A plurality of validation frameworks abound including data annotations, FluentValidation, NHibernate Validators, Enterprise Library Validation Block, etc. Validation frameworks however, can be abused because one can be lead into thinking that a framework solves all validation concerns, across all application layers. Unfortunately, this is not always possible. In practice, I’ve found that validation frameworks are best suited for use at application layer boundaries - such as validation user input in the presentation layer, ensuring database constraints at the the persistence layer, or enforcing conformance to a schema in a REST adapter. Implementing validation at each layer separately allows validation to be context specific. However, this can lead to a degree of duplication, in response to which DRY fanatics will scream blasphemy. This subject is addressed in the next section.
The domain layer is best kept lean with use of plain-old-exceptions to enforce validation rules. This is because validation frameworks carry a requirement to invoke the application framework, similar to the “IsValid” methodology addressed above. A presentation layer developed with ASP.NET MVC provides action handlers to inject validation invocation logic. In this way, a validation framework can be applied globally toward the entire application. Similar injection points don’t exist in plain C# or Java code, and use of an AOP framework can add needless complexity. This programming language “short-coming” can be overcome with extensions to homoiconicity. [.NET Code Contracts] can be regarded as a validation framework with extended static verification. The more general paradigm of Design by contract proposes that software should be written in terms of formal and verifiable specifications. The Eiffel programming language, the creator of which gave use [command-query separation] (http://bit.ly/oAo2b) among other things, is based on this principle at the its very core.
A Stackoverflow question asks whether there is a way to re-use validation logic. In practice, it is often simpler to allow a degree of duplication rather than to strive for complete consistency. Consider the following example. Suppose we have a customer entity where the customer’s name is required. Enforcing this constraint in the entity is trivial:
An entity never stands alone however and we must consider the clients of this class. Who calls the constructor? In a web application implemented using ASP.NET MVC there would be a corresponding customer view model. This view model is part of the presentation layer and is designed with data binding in mind. Data annotations can be used to declare validation rules:
This class is very similar to the domain entity class but with several important differences. It has a parameterless constructor, a name property without a guard clause and a data annotations validation attribute. These aspects of this class make it suitable for use in the presentation layer. Instead of attempting to carry validation rules from the entity class, the name requirement constraint is effectively duplicated. This is indeed duplication, but it must much simpler to manager than a some sort of validation mapping framework.
Entities can enforce certain invariants but the scope of these invariants are always limited by the entity itself. Since entities should be lean and self-contained, without access to external services or repositories, they may not have access to the resources required to enforce certain validation rules. In this case, the application service can serve a mediating role and procure the resources required to enforce validity. There exist business rules that are not natural responsibilities of an entity or a validation framework. For example, a uniqueness constraint on user’s user name cannot be verified by an entity because the entity does not and should not have access to the database of existing users. Instead, this rule can be enforced in the application service. Furthermore, rules may become sufficiently complex to warrant a business rules engine, in which case the application service is once again tasked with enforcing validation. An even more ambitious discipline is ontology engineering where CycL is an ontology language. Ontological engineering purports to formalize all business rules in machine executable representations. Jeff Zhuk, one of the leading practitioners in this field, proposes a Knowledge Driven Architecture based on these technologies.
Although most queries return an object or a collection of objects, it also fits within the concept to return some types of summary calculations, such as an object count, or a sum of a numerical attribute that was intended by the model to be tallied.
This particular responsibility is not well explored or developed and as a result is often overlooked in practice. The basic premise of this approach is to leverage the underlying database, a relational database in particular, for what it’s good at - namely aggregation, filtering, and projection. The requirement for this type of summarizing information depends on the project, but is commonplace in applications involving any type of GUI.
Consider the standard order model with an order entity containing a collection of line items. The order total is a sum of the constituent line item totals and the tax and shipping charges. In C# the corresponding classes might look something like this:
Note that much of the behavior and constraint checking have been omitted for brevity.
This model appropriately represents the domain at hand. The only real behavior on this model is the calculation of the total. A typical application requirement calls for a view that displays a list of recent orders showing the order number, date and total. To support this, a repository contract might look like this:
From a client perspective the repository contract satisfies the requirements. However if implemented using an ORM such as NHibernate certain caveats arise even in this simplistic scenario. The relationship between an order and an order line item is one-to-many and in a relational model there would be a table for the orders and a table for the order line items related by a foreign key. When retrieving an order entity, the repository must also retrieve the order line items in order for the entity to be complete. This can be achieved using several fetching strategies, including joining the order line items table or issuing a second select statement to the database. For retrieving a single instance of an order entity this approach is acceptable however for collections of orders, as in the case of the GetMostRecent method on the order repository, this results in the select N+1 problem. While NHibernate and other ORMs offer solutions to this problem, these solutions are relatively complex and can become a performance burden for both the database and the ORM. An alternative solution, one that is accordance with KISS, is to create an aggregating database query and map the results to simple, read-only objects:
One could also use the projection facilities provided by NHibernate. Regardless, the target class should match the shape of the query:
Note that this class contains no behavior and is read only. Instances of this class are meant for read-only purposes such as for display in a GUI. Accordingly, these types of classes are called read-models. Alternative names for these types of objects are reporting objects, views, projections and probably something else I’m not thinking of at the moment.
This trivial pattern boasts several advantages. The first of course is performance since relational databases are very adept at these types of queries. Next is simplicity - there is no simpler way of pulling data from a database - no ORM, no mapping magic, no problem! Furthermore, the developer isn’t faced with coercing the order entity class to match the shape of the query at hand. This is perhaps the central advantage from the DDD perspective - keeping the aggregates and entities pure and Occam-esque. Recent DDD guidance suggests that aggregates should reference other aggregates using only the identity instead of direct association. This also simplifies the model and transitively the mappings. What if, however, a particular view requires the display of data contained in the associated aggregate? The aggregate itself can no longer be used but this is where the read-model pattern comes to the rescue. Conversely to the read-model, the write-model aggregate only need to be retrievable using its identity in which case an ORM is a suitable solution due to several convenient accommodations such as change tracking, generated SQL, concurrency control, etc.
The read-model pattern can be compared to the notion of consumer-driven contracts. Additionally, the read-model pattern can be viewed as a special case of CQRS in that the model used for querying is different from the model used for processing commands. This is not full-fledged CQRS because read-models are pulled from the same data source as the write model. There is no requirement for a synchronization mechanism and the affiliated consistency and cache control concerns. However, the option remains to implement the synchronization mechanism at a later time when performance tests clamor for it. In CQRS, these read-models are usually called persistent read-models and are generated by event projections. The CQRS foreshadow also positions read-models as an intermediary refactoring step toward a CQRS implementation. For in depth treatment on CQRS look no further than Rinat Abdullin.
]]>The term service is overloaded and its meaning takes on different shades depending on the context. As a result, there is a cloud of confusion surrounding the notion of services as one tries to distinguish between application services, domain services, infrastructural services, SOA services, etc. In all these cases the term “service” is valid, however the roles are different and can span all layers of an application.
A service is indeed a somewhat generic title for an application building block because it implies very little. First and foremost, a service implies a client the requests of which the service is designed to satisfy. Another characteristic of a service operation is that of input and output - arguments and provided as input to an operation and a result is returned. Beyond this implication are usually assumptions of statelessness and the idea of pure fabrication according to GRASP.
Eric Evans introduces the notion of a service as a building block within Domain-Driven Design in the blue book:
When a significant process or transformation in the domain is not a natural responsibility of an ENTITY or VALUE OBJECT, add an operation to the model as standalone interface declared as a SERVICE. Define the interface in terms of the language of the model and make sure the operation name is part of the UBIQUITOUS LANGUAGE. Make the SERVICE stateless.
These types of services can be identified more specifically as domain services and are part of the domain layer. Domain services are often overlooked as key building blocks, overshadowed by focus on entities and value objects. On the other end of the spectrum is over-utilization of domain services leading to an anemic domain model in what essentially becomes a separation of data, stored in entities, and behaviors, provided by the service. This can become an anti-pattern because the information expert aspect of OOP is lost.
Domain services are different from infrastructural services because they embed and operate upon domain concepts and are part of the ubiquitous language. Infrastructural services are instead focused encapsulating the “plumbing” requirements of an application; usually IO concerns such as file system access, database access, email, etc. For example, a common application requirement is the sending of an email notification informing interested parties about some event. The concept of the event exists in the domain layer and the domain layer determines when the event should be raised. An email infrastructure service can handle a domain event by generating and transmitting an appropriate email message. Another infrastructural service can handle the same event and send a notification via SMS or other channel. The domain layer doesn’t care about the specifics or how an event notification is delivered, it only cares about raising the event. A repository implementation is also an example of an infrastructural service. The interface is declared in the domain layer and is an important aspect of the domain. However, the specifics of the communication with durable storage mechanisms are handled in the infrastructure layer.
An application service has an important and distinguishing role - it provides a hosting environment for the execution of domain logic. As such, it is a convenient point to inject various gateways such as a repository or wrappers for external services. A common problem in applying DDD is when an entity requires access to data in a repository or other gateway in order to carry out a business operation. One solution is to inject repository dependencies directly into the entity, however this is often frowned upon. One reason for this is because it requires the plain-old-(C#, Java, etc…) objects implementing entities to be part of an application dependency graph. Another reason is that is makes reasoning about the behavior of entities more difficult since the Single-Responsibility Principle is violated. A better solution is to have an application service retrieve the information required by an entity, effectively setting up the execution environment, and provide it to the entity.
In addition to being a host, the purpose of an application service is to expose the functionality of the domain to other application layers as an API. This attributes an encapsulating role to the service - the service is an instance of the facade pattern. Exposing objects directly can be cumbersome and lead to leaky abstractions especially if interactions are distributed in nature. In this way, an application service also fulfills a translation role - that of translating between external commands and the underlying domain object model. The importance of this translation must not be neglected. For example, a human requested command can be something like “transfer $5 from account A to account B”. There are a number of steps required for a computer to fulfill that command and we would never expect a human to issue a more specific command such as “load an account entity with id A from account repository, load an account entity with id B from account repository, call the debit method on the account A entity…”. This is a job best suited for an application service.
The following is an example application service from a purchase order domain. The example also contains a PurchaseOrder aggregate, an Invoice value object and a repository. (Please note that the code has been simplified for explanation purposes).
There are many refinements that need to be made to this code for it to be of production-ready caliber however it serves well to illustrate the purpose of an application service. The interface IInvoiceNumberGenerator is indeed a domain service because it encapsulates domain logic, namely the generation of invoice numbers. This process is something that can be discussed with domain experts and users of the system. After all, the purpose of the generator is to make use of invoice numbers of palatable. By contrast, the PurchaseOrderService application service performs technical tasks which domain experts aren’t interested in.
The differences between a domain service and an application services are subtle but critical:
In a complete application, a domain model does not stand alone. Applications with GUIs contain a presentation layer which facilitates interaction between the domain and a user. The same application may wish to expose its functionality as a set of REST resources or WCF services or SOA services. In Alistair Cockburn’s Hexagonal Architecture, the presentation layer, the REST resource and the WCF service are adapters which adapt the core application to specific ports. Application services form the API which encapsulate the application core and in the case of Domain-Driven Design they ultimately orchestrate and delegate to the underlying entities, value objects and domain services. The application service isn’t strictly necessary since each adapter implementation can orchestrate the required domain elements, however encapsulating the domain layer provides a fitting demarcation allowing each component of the entire application to be viewed in isolation. Conversely, the DDD-based domain layer isn’t strictly necessary and the application service could delegate to a transaction script. As seen from this perspective, DDD is an implementation detail.
]]>