On Message Queue Technologies

The Database As Queue Anti-Pattern
“Why is it an anti-pattern?
1) polling either short interval hammers the db, or long interval makes system irresponsive.
2) polling queries should be fast, need to implement index, which makes inserts slow, contradiction.
3) clear records to make the table efficient, usually deletes are slow
4) sharing a database between applications creates hugely coupled monster.

use a messaging system. no more polling, efficient message delivery, no need to clear completed messages from queues, and no shared state.”


Reliable Delivery Pub/Sub Message Queues with Redis

ActiveMQ, Apollo, Beanstalkd, Celery Crossroads I/O, Darner, Delayed::Job, Gearman, HornetQ, IronMQ, Apache Kafka, Kestrel, nanomsg, NATS, NSQ, Apache Qpid, queue_classic, RabbitMQ, Resque, RestMQ, RQ, Sidekiq, SQS – Amazon Simple Queue Service, Zaqar (ex Marconi), ZeroMQ, huey,


Broker vs. Brokerless ZeroMQ


Queuing Windows Services work items using MSMQ, and live progress using WCF Net Named Pipe


Messaging patterns
messaging patterns presentation


Messaging Design Pattern and Pattern Implementation
“Encapsulation. The messaging design pattern maximizes encapsulation. As mentioned earlier, each
component is a self-contained/independent unit. The only mechanism of communication with other
components and applications is via messaging.
Decoupling. MDP minimizes coupling. Again each component is a self-contained unit that can perform
independently from the rest of the system. The communication mechanism is decoupled from component
Reusability. MDP improves reusability. This is similar to the building blocks in a “Lego” set. Very complex
models can be built based on simple pieces that share a simple way of interconnecting them (i.e. common
interface). The power of the approach is derived from the number of combinations in which these toy
pieces can be assembled. Components that use MDP can be interchangeably plugged into complex
applications. The components can be assembled in a limitless variety of configurations. The user of a
component only needs to know the input/output messages that the component handles. Applications are
also able to reuse components from other applications at the component level: a single component can be
extracted from another application, provided that the messaging design pattern is being used.
Scalability. During the review of this paper, it has been proposed that MDP can be applied to improve
scalability (see acknowledgements). Because of tight coupling between client and server, conventional
distributed/service technologies based on RPCs require that client and server application be upgraded at
the same time. This is usually not feasible and presents significant scalability limitations for infrastructures
running 24/7 and/or expecting to handle a large number of computer nodes. On the other hand, MDP does
not present this limitation because client component, server component, and communication mechanism
are decoupled. Servers can be upgraded one by one without an impact on the client application and the
rest of the infrastructure. Once all the servers have been gradually upgraded, clients can be upgraded to
take advantage of the new software functionality. As a consequence, an infrastructure based on MDP can
scale well and handle an arbitrary number of servers and clients 24/7. This MDP application assumes that
the new software version is backward compatible.
QA/Testing process. MDP facilitates testing and debugging efforts. Components are tested as independent
units by sending messages to the component and verifying the expected reply messages (black-box
testing). Keep in mind that all the components share the same messaging interface. In general, unit testing
can be performed via a testing harness. No need to include testing code inside the component code which
can be time consuming and lead to the unexpected introduction of software defects.
Design process. MDP improves and simplifies the design process. The bulk of the design work becomes
defining the set of components needed to meet the system requirements and the input/output messages
that each component needs to handle. There is a tight correspondence between UML design diagrams and
the components needed for the implementation. Several UML diagrams are geared towards messaging
(sequence and collaboration) although their implementation does not rely on it. The UML model and the
implementation are disconnected when MDP is not used. Since all components share the same messaging
interface, they can also be readily added to BPM/BPEL diagrams. As mentioned earlier, this is similar to
building blocks that can be reused and connected in many different ways.
Development process. Since each component that relies on messaging is self-contained, a large team of
people can cooperate in the development effort without stepping on each other’s work/code. In the ideal
situation, responsibility for one component/package can be given to an individual. The rest of the team only
needs to know the input/output messages that someone else’s component is designed to handle. In
general, there is no need to change someone else’s code. The need for creating, maintaining and merging
several versions of the code is also minimized or eliminated. Testing/QA engineers can perform their
testing independently via a testing harness. As a general rule, there is no need to add testing code to the
component itself.
Logging and Debugging. Since all the components use the same messaging interface, messages can be
logged automatically. This minimizes the need for print/logging statements inside the code which can be
time consuming and error-prone. By taking a look at the messages being interchanged and automatically
logged, the user is usually able to quickly track down the message/component that is causing the problem
(with minimum or no extra effort).
Security. Well-known encryption and authentication mechanisms fit in well with the messaging design
pattern. Strong security can be provided by the framework that implements MDP [3]. This is done by
encrypting and authenticating the messages being interchanged. The sender and the recipient do not need
to be too concerned with how secure messaging is implemented. This provides strong security while at the
same time simplifying the implementation of security. If required, custom security mechanisms can also be
incorporated: sender and receiver need to agree on, and implement the message encryption/authentication
mechanism to be used.
Multithreading and asynchronous messaging (Live or Animated objects). MDP is able to handle the
complexities associated with multithreading and asynchronous messaging. Components that implement
MDP are able to execute in a separate/independent thread. This is a natural representation of the real
world: each component (entity) is a self-contained unit and executes independently for the rest of the
system. Messages can be processed asynchronously using the component’s own independent thread.
Entities that follow this pattern are referred as live or animated. This capability is usually implemented in
the context of a component framework [3]. The component does not need to add separate logic to handle
multithreading which is time consuming, complex and prone to error. ”


Messaging Patterns in Service-Oriented Architecture, Part 1
“This first of a two-part series discusses how messaging patterns exist at different levels of abstraction in SOA. Rather than explicitly declaring how systems will interact through low-level protocols and object-oriented architectures, SOA provides an abstract interface through which a service can interact with other services or applications through a loosely coupled (often asynchronous), message-based communication model. (23 printed pages)”

Messaging Patterns in Service-Oriented Architecture, Part 2
“Explores the contract patterns that illustrate the behavioral specifications required to maintain smooth communications between service provider and service consumer; also explores message construction patterns that describe creation of message content that travels across the messaging system. (9 printed pages)”



Chronicle Queue
“Chronicle Queue works like a standard Java Queue, but unlike the standard queue, Chronicle Queue is able to persist every single event, It records each event to a memory mapped file.”