Architecture

Sodacan is designed as a collection of microservices with an infrastructure to support them. Decision-making microservices in Sodacan are called Modules. Microservices that interface with external systems and devices are called "Message Adapters". In general, Sodacan microservices have the following characteristics:

Message Oriented
Independently testable and deployable
Loosely coupled

Additionally, Sodacan modules are:

Domain-specific
Declarative
Friendly to non-programmers

Message adapters are technical components that - gather input, create messages, and put them on the message bus, or - send messages from the bus onto external systems

The technology used by an adapter varies by adapter, but on one side of the adapter is usually the Sodacan message bus. The other side being whatever behavior is needed to interface to message, database, or similar.

Message Oriented

A message oriented architecture, if taken to its extreme, means that there is a complete decoupling between components. Sodacan attempts to do just that. The cost of a message oriented architecture is that there is an overhead to using messages. An actual message is created by a producer, published to a broker (message bus) and subscribed to by zero or more consumers. In Sodacan terms, modules, which provide the decision making, and adapter modules, which interface to the outside world, are the end-nodes in a message-based system. A nice side-effect of this design is that the overall data-flow is relatively flat: components send messages to or receive messages from the message bus. The only minor exception is that some components may use local IO to access a datastore for persistence.

Components of this system communicate using publish/subscribe semantics. You should be at least a little familiar with publish-subscribe design pattern before reading further. In short, publishing a message means making it available to an subscriber interested in the message. The publisher need not be concerned about the destination of a message. The dataflow though a component is always the same:

flowchart LR;
    A[Message Bus] -. subscribe .-> B[Component];
    B[Component] -. publish .-> A[Message Bus];

Module Testablility

Modules are 100% standalone with no dependencies on any other modules. Knowing this, the author of a module should not need to be concerned with anything other than what messages that module receives and what messages it produces. And, because it is message oriented, there is no restriction on where messages originate from (or where they go). A module does not need to be "wired up" at any time.

To unit test a module only requires a collection of messages to be fed to the Sodacan runtime and a way to verify that the resulting messages, if any, contain the expected results. The message source will contain the module itself as well as any input messages.

With the use of "deployment mode", a module can be integration tested in a "live" environment with no effect on the real live environment.

Messages

In Sodacan, PUBLISH variables are essentially messages waiting to be sent. And, SUBSCRIBE variables are messages waiting to be received. Messages are exchanged through what is called a topic which is defined in more detail below. Simply put, a topic groups together messages of a specific format. That format is then the topic name.

All messages contain a timestamp which implies a temporal sequence for messages. Messages contain an "offset" attribute which uniquely describes each individual message within a topic.

The producer is also identified in a message. Messages also contain a key and a payload, both of which are optional.

Message Bus

Abstractly, a message bus exits to exchange messages. Ignoring security, anyone can produce a message and anyone can consume messages. In Sodacan, the message bus is an implementation detail handled in the background. The modules that make up a system are unaware of the bus itself. Like a post office handles the logistics of getting a newspaper from its source (producer) to its destination(s) (consumer(s)). In a message bus architecture, the producer of a message as no control over who consumes that message. And, in general, the consumer has no control over who, how or when the messages it receives are produced. This is the essence of decoupling in a microservice architecture.

In the following diagram, messages are produced by Modules A and C without any knowledge of where they will be consumed or even if they will be consumed.

flowchart BT;
    A[Module A] -. publish .-> B[Message Bus];
    C[Module C] -. publish .-> B[Message Bus];

When Module D is added, it can consume messages from the message bus as it sees fit.

flowchart BT;
    A[Module A] -. publish .-> B[Message Bus];
    C[Module C] -. publish .-> B[Message Bus];
    B[Message Bus] -. subscribe .-> D[Module D];

Message Producer

A MODULE that contains one or more PUBLISH statements is a message producer. Each PUBLISH variable is sent onto the message bus.

Message Consumer

A MODULE that contains one or more SUBSCRIBE statements is a message consumer. A module is only able to "see" the information it receives via message (or the passage of time). In Sodacan, there is no such thing as peeking into another module to find a value. So, it is important to ensure that information needed by a consumer arrives via message.

Topology

The underlying messaging technology will determine the topology of a working Sodacan installation and how application components are deployed. The following table show the application components and where each resides in different configurations:

Stand-alone

A Sodacan stand-alone configuration is useful for simple demonstrations and some development. It uses no communication except for a web server and it would be difficult to connect to real devices. Persistence is periodically stored in a flat file in JSON format. No reliability guarantee of the data. Communication between modules and the message bus all occur in a single address space. Keep in mind that performance in this simple configuration will be very fast. But it won't scale well. Don't try to draw conclusions about performance from this configuration.

Single-node

A Single-node configuration uses Apache Kafka in a more-or-less real configuration but has no redundancy and does not scale. All of the components are the same as a distributed configuration though communication between components may still be faster because they are all on the same node. For a smaller installation without the need for high-availability and fault tolerance, the single-node configuration may be sufficient. Almost all unit and integration testing will work fine in this configuration. The part that isn't covered will be specific to Kafka.

Distributed

A distributed configuration also uses Apache Kafka but with multiple brokers and topic partitioning. This configuration provides the highest throughput, scalability and best reliability. The transition from single-node to a distributed configuration is possible without having to start-stop the system. However, it does require careful planning and execution.

System Components

---
title: Dependencies
---
flowchart TB;
subgraph application
  wp(Web Pages)
  dc(Dev<br/>Controller)
  style dc fill:#f9f,stroke:#000
  ua(User App)
  style ua fill:#f9f,stroke:#000
  cli(Command<br/>Line<br/>Tool)
  ag(Module Agent)
end
subgraph IO
  log(Logging)
  mod(Mode)
  ss(State<br/>Store)
  mb(Message<br/>Bus)
  cl(Clock)
end
ap(Sodacan API)
co(Compiler)
ka(Kafka Admin)
m(Module)
oa(O/S Admin)
ra(REST API)
rt(Runtime)
ws(Web Server)
cli --- ka
cli -.Sodacan Admin.- ap
cli --- oa
ag --- rt
ap --- mod
rt --- mod
rt -.Execute.- m
rt -.Compile.- co
mod --- cl
mod --- mb
ra --- ws
ws --- ap
ua --- ap
dc --- ra
dc -.alternate.- ap
wp -.static.- ws
wp --- ra
mb --- k
cl --- rtc
cl --- tc
mod --- ss
ss --- mem
mb --- mem
ss --- f
mod --- log
log --- f
log --- mem
subgraph plugin
  f(File)
  k(Kafka)
  tc(Testing<br/>Clock)
  rtc(Real Clock)
  mem(Memory)
end

Runtime Behavior

An API call initiates module processing in one of several threads involved in the processing of messages. Most of the remainder of this section is under the control of the message bus which calls back to a module runtime. Put simply, the message bus will notify the runtime factory that a specific module is active allowing the runtime to fetch and compile the module's source code and setup structures needed for that module. The message bus will then feed messages to that runtime as they arrive.

Each module runtime will be called from a separate thread but messages for any one module will be delivered sequentially.

The following diagram shows the key interactions between the API, the message bus, the runtime and runtime factory.

---
title: Runtime Flow
---
sequenceDiagram
    autonumber
    participant A as API
    participant RF as Runtime Factory
    participant R as Runtime
    A-)MB: Start Agent
    A-)MB: Add/update module
    MB->>+RF: Module Active with Initial State
    RF->>R: Create Instance
    R-->MB: Get Module Source
    R->>MB: Subscribe Topics
    par Thread Per Module
    MB->>+R: Process Event
    R->>MB: Save State, Publish Variables
    end
    MB-)R: Module Inctive

An API call gets things started by establishing a Sodacan Agent. The message bus starts running in a separate thread so control return immediately. Modules will begin processing soon after the agent gets started.
The application is then free to add, change or delete modules using the API. This can be done at any time before or after the agent is started. The module changes become part of the message flow which will cause the runtime to be updated when the changed module arrives.
The message bus (agent) will then make a call to the runtime factory whenever it determines that a message has arrived for that module yet no runtime yet exists.
The runtime factory creates an instance of the runtime bound to that one module. The runtime will remain active for that module until deactivated.
When the runtime for a module starts up, it will need to call back to the message bus to get the source code for that module, compile it, and create any structures that it needs.
The runtime will also need to tell the message bus what topics it subscribes to. The message bus will subscribe to each of these topics plus the tick source plus module state messages.
At this point, the Message Bus will deliver one message at a time to the runtime's processMessage method. The runtime will execute the module using the incoming message containing an event.
When done, the runtime will save the state of the module's variables by sending them back to the message bus. This makes the variables available to other modules that have subscribed to them.
When the message bus determines that the module is no longer active, it calls back to the Runtime which can then release any resources that it may have used.

Message Bus Behavior

While it may appear that the message bus runs all modules in a single address space, that is not usually the case. For the Kafka plugin, here's how it actually works. In Sodacan, a given module can only be executing in one place at a time and processing a single stream of events. If there is only a single "agent" running, then in fact all active modules will be in memory at the same time.

However, if two or more agent's are running, then each agent gets a share of the modules. If more agents are running than there are modules, then the extra agents will be running idle. The extra agents can be though of as hot backups since Sodacan (Kafka broker) can redirect traffic to the waiting agent at any time. If a new module is added to the system, this also affects what is running where. The message bus (Kafka) is free to "rebalance" the load at any time meaning the runtime and runtime factories must respond to the activation and deactivation callbacks.

Due to the nature of Kafka, subscriptions to multiple topics are not merged and delivered in timestamp order. (Within a topic, order is guaranteed). Therefore, Sodacan adds an intermediate "priority queue" so that as events are received, they are added to the queue which is ordered by timestamp and then delivered to the module's runtime. The ordering issue only comes up when processing is stopped and therefore has to "catch up".

Event Processing Detailed Description

The runtime for a module is fed a never-ending stream of records.

Type of record	Description
a-scc	Save the source offset id in current state. Don't compile yet
m (for this module)	The module state can be ignored except that if the source code offset has changed, the source code should be fetched and compiled. Also, the variables can be saved as state in the runtime.
m (for subscriptions)	Determine which attribute changed and treat it as an event. Ignore offsets that are not greater than the last offset processed for the given variable.
e	This is an explicit event
a-stop	The runtime should close any resources it is using. No further messages will be delivered to this runtime.

Topic Allocation

Each module gets its own topic. That topic will contain one message per processing cycle. This topic is what the module uses to store state and it is also what other module subscribe to if that module is interested in the results of this module.

Another topic is used to maintain a list of all modules.

There is a single topic containing all versions of all module's source code. Only the Kafka offset for a particular version of a module is referenced in the module's state. This keeps the state record small. When the runtime sees a change in the offset for the module source code, it will fetch the new module and compile it. As with all other activity related to a module, this compile happens between events.

Module Origination

The source code for a module begins on the file system. After initial introduction to Sodacan, it can be extracted from the message bus and updated. In any case:

Compile the module, this yields the module and a newly minted collection of variables known to that module.
The variables for a module have a different lifecycle than the module source code. For example, variables are unaffected by a source code change.
A topic is created for each new module. If the module already exists, the topic is unchanged.
This topic may already exist if some other module subscribed to it. But the topic won't contain any records.
Conversely, this module will create any topics needed by its subscribe variables.
The module's state topic is updated after each processing cycle for that module.
Therefore, the last record in the topic contains the most recent state of the module. That is, offset minus one.

Sodacan API

The Sodacan API provides a way to perform administrative, operational, and application functions. Many of its functions are passed through messages to other components including the Sodacan web server, the underlying Kafka system, and mostly to Sodacan agents. The API in Sodacan is separate from the RESTful API in the web browser. Both provide similar capabilities but the Sodacan API talks directly to the message bus whereas the RESTful API is, of course, HTML-based which in turn uses the Sodacan API. The RESTful API is useful when the Sodacan message bus is behind a firewall or where it is more practical to use an HTTP-based solution.

Command Line Tool

The Sodacan command line tool provides a number of administrative functions including stating and stopping the server(s), creating topics, compiling and deploying modules, creating and managing modes, etc. It uses the Sodacan API.

Web Server

The web server provides the same capabilities as the command line tool but in a graphical format. It also includes a dashboard for monitoring a running system. It uses the Sodacan API. The web server is also what exposes the Sodacan RESTful API. Sodacan uses static web pages, which it serves, which in turn call the same APIs which remote applications can use independent of web page, subject to authentication and authorization.

Message Bus

The Message Bus is a wrapper around Apache Kafka. Kafka is accessed only through Kafka's client APIs. An existing Kafka (and Zookeeper) installation can be used if necessary. A docker-based Kafka installation is also usable but be certain to externalize the message storage. The message bus wrapper (in Java) is needed to support the stand alone configuration and for unit testing Sodacan core code. It also allows plugin of an alternate message bus although no such plugins are available in Sodacan, yet.

The message bus in Sodacan is responsible for reliably storing messages for however long is needed. This is the primary means of storage in Sodacan. Messages are the official "source of truth" in Sodacan. The other data stores such as module persistence can be recovered by replaying messages. When the Message Bus is Kafka, each Kafka broker stores these messages close to where the broker is running. If Kafka is running with replica > 1, then there will be multiple copies of messages on different brokers.

If the stand alone configuration is used, then messages are not stored reliably.

Module Agent

Module agent(s) are the workhorse of Sodacan. These agents host one or more modules and provide the timer, clock, persistence, and interface to the Message Bus.

Logging Agent

Component	stand-alone	Single Node	Distributed
Command Line Tool	The entire system runs inside the tool.
WebServer	N/A
MessageBus	N/A
LoggingAgent	N/A
ModuleAgent	N/A
ModulePersistence	N/A
LoggingAgent	N/A

The smallest configuration will have a single agent that runs all modules in all modes in one memory space.

Module Topic Structure

Each module has it's own topic. More specifically, topics are named as follows:

Component	Description
"m-"	Prefix to identify all module state topics
module	The module name
"-"	Separate module name from mode name
mode	Deployment Mode. Not to be confused with any variables that happen to be named mode.

Message Format

Messages are organized by topic as described above. Within a topic, individual messages contain these fields:

Field	Location	Description
Offset	internal	A permanent incrementing non-repeating integer within the topic
Timestamp	internal	When the message was published
Producer	internal	The name of the producer (eg module) that gave rise to this message
mode	key	Deployment Mode. Not to be confused with any variables that happen to be named mode.
domain	key	The full domain name of the local Sodacan broker
topic	internal	The original topic name may be the same as producer, and is usually the same as the name of the topic containing this message
instance	key	The module's instance, if any
variable	key	The variable (or event name)
value	value	The value of the variable, if any

Message Delivery

When a message is published, it is immediately available to any subscribing modules, baring hardware or infrastructure difficulties. If a consumer (module) is unavailable, the message will be delivered when the component is restored.

Latency between a message being published and being consumed should be in the neighborhood of 1-20 milliseconds, depending on the underlying hardware. Any application that depends on faster delivery should seek another solution.

Module Persistence

Since events arrive at a module one-by-one, it is important to maintain state within a module. For example, a lamp module might have a "mode" setting that determines how other messages are handled. The mode-setting message will have arrived sometime before subsequent messages are processed that need the value of the mode setting. In the following, the mode variable will have been set via message some time in the past. When midnight arrives, that variable will be needed. Between those two times, the module may be off-line (crashed, power failure, explicitly taken off-line, etc). So, when the module needs to be restored, the variables must also be restored.

    MODULE lamp1
        SUBSCRIBE mode  {off, auto, on} 
        PUBLISH state {on,off}
        AT midnight       // Turn off this light
          AND mode.auto  // at midnight
          THEN state=off  // if mode is auto

Persistence is handled automatically by Sodacan. At the conclusion of processing an event (a cycle), all of the module's variables are sent to the module's topic on the message bus. This "module snapshot" serves several purposes:

A snapshot of the modules variables which can be recovered quickly
The offsets into the subscribers message state topics
Exposes the "publish" variables belonging to this module that can be "subscribed" to by other modules.

The snapshot record is composed of the following:

A message ID (offset) uniquely identifying this snapshot within the module's topic.
A timestamp of the snapshot
The record "key" indicates the type of record (see below)
The mode and module name (this is implied by the topic)
The name and value of each of the publish and local variables used by the module
The name and value of the inbound event (subscription) giving rise to this cycle
The topic name and offset of each of the modules subscribed to by this module
The messageId (offset) of the source code for the module responsible for processing this cycle

The record key indicates a "type" of record. The primary type is snapshot as described above. The other type of record is an admin record.

"s" = snapshot
"a" = admin
"e" = Simple event

The admin record type is further subdivided with only one option at this time:

"a-scc" in the key with the value containing the offset into the Modules topic containing scc source code.

Here's how the source code is deployed. When a new version of a module is compiled and then deployed, the source code is sent to the ModuleSource topic. The offset is acquired for this instance of the source code. A message is then send to the module's snapshot topic with a key of 'a-scc' and a value containing just the offset into the module source code topic. The module's runtime will then see this special admin record, fetch the source code from the specified offset, and compile it. This has a very nice effect: the point at which a module was changed is preserved in the message stream. In other words, a full audit trail is created. It also means that there is no need to manually deploy new modules as they are created or modified. The flow of messages into a module might look like this:

flowchart TB;
   subgraph MB[Message Bus]
        direction LR
        a[variable a] -.-> 
        b[variable b] -.-> 
        c[module update] -.-> 
        d[variable c];
   end
    MB --> MA
   subgraph MA[Module Agent]
        direction TB
        e[module]<-- Save/Restore -->ds[data store]
    end

An e-type of record is similar to a snapshot but only contains The name and value of the inbound event, not the rest of the state of that module. This type of record is used by device interfaces which only populate events (buttons, sensors, etc). They don't have state.

Each local or publish value contains a version. This is important since that will be the way a runtime can determine that a value "happened" ie it's an event vs just saved along with other state. Specifically, when a module sets the value of a variable, the version number is increased by one. The runtime subscribing to a given state looks for a version change in received states to determine which, if any, variable is a new event.

Bootstrapping a variable version is important. The first time a particular variable is received by the runtime, it can;t absolutely determine if the state is an event or not.

A curious aspect of the module's topic is that the runtime for the module publishes to this topic but the runtime also must subscribe to the same topic along with other topics that the module is interested in. Normally, the runtime can ignore most of this self message. However, there are at least two cases when this self-message is useful:

If the id of the source code has changed since the last event was processed which will cause the runtime to load a new version of the module
If the runtime has lost it's state (or never had it), the state can be recovered from this message. This make fault tolerance possible because a failure in one agent can be quickly recovered in another agent without data loss.

Another thing that the runtime for a module must ignore are messages that are older than the offset kept for that topic. This happens because the message bus could be serving the same event to more than one module at the same time but one module may lag behind other modules in processing. Since the message bus does not inerpret the contents of events, it is up to the runtime to see to this housekeeping detail.

Infrastructure

Module deployment

The Sodacan command line interface provides all the information needed to load and compile modules and to start and run module agents.

Web Application

The Sodacan web application has several top-level windows:

User Account - maintain the user's account, login page, etc
Administration - Create users, maintain topics,
Operations - Monitor message bus storage and message traffic
Application Console - Modules, buttons, etc

See Web Server for more details.

Comparisons to Conventional Approaches

Modules can be thought of as Java/C++ class definition but in reverse. The term "static" is used to distinguish class-wide variables whereas Sodacan makes variables without any indication otherwise, a static. Conversely, when referring to an instance variable, Sodacan requires what may look like an array reference to instance variables.

Module Persistence in Sodacan is not unlike systems such as Apache Flink which, like Sodacan, stores persistent data with the end-point rather than having to connect to a database. This approach, along with messaging, virtually eliminates the need to deal with database concurrency, locking and similar problems.

The module language is line oriented, similar to Python but without its indent sensitivity. Sodacan module language is case sensitive.

Aliases are used in modules and they look like a SQL alias:

    SELECT primaryPhoneNumber AS ppn FROM ...

In a Sodacan Module

    MODULE lamp1
        SUBSCRIBE aModuleName.verylongvariablename AS shortName
        ...

They also work the same as in SQL and follow the expression or variable as in SQL.

The module language is closer to a domain-specific language than a true programming language for several reasons:

It has very little technical chatter. Only a few very broad data types. No such thing as int, int_32, BigDecimal, etc. Just a number (Sodacan uses the term NUMBER).
The hierarchy is shallow. There are modules and variables within modules and that's about it. Traditional IF statements and code blocks (begin-end-style) are not used to keep the module shallow. This is similar to the way some rule languages control the depth of statements.
Invocation is different from traditional programming languages. No such thing as a function or method call. When a message arrives in a module, it is immediately made available in a variable. This activates the module for one cycle. The module then waits for the next message to arrive. This means that open/close/loops etc are unnecessary.

The THEN statement might seem novel. If there is more than one thing to do as a consequence of a conditional expression, then the THEN is repeated.

        ...
        THEN do the first thing
        THEN do the second thing

This extra bit of typing (THEN) eliminates the need for expression separators such as a semi-colon.

Architecture

Message Oriented

Publish Subscribe

Module Testablility

Messages

Message Bus

Message Producer

Message Consumer

Topology

Stand-alone

Single-node

Distributed

System Components

Runtime Behavior

Message Bus Behavior

Event Processing Detailed Description

Topic Allocation

Module Origination

Sodacan API

Command Line Tool

Web Server

Message Bus

Module Agent

Logging Agent

Module Topic Structure

Message Format

Message Delivery

Module Persistence

Infrastructure

Module deployment

Web Application

Comparisons to Conventional Approaches