Joule decentralizes signal processing into discrete modules. These modules are connected by streams as shown in the figure below. The interconnection of modules and streams form a data pipeline. A pipeline may execute as a single proces, a collection of processes, or even be distributed across multiple nodes in a network without adjusting any module code.
Joule is a system service. Use the
joule command to interact with the service.
Joule constructs the pipeline based on configuration files in /etc/joule. Details on these configuration files is provided in the sections below.
Modules are executable programs managed by Joule. The module configuration format is shown below:
Module configuration files must end with the .conf suffix and should be placed in /etc/joule/module_configs. See the list below for information on each setting. Only the [Main] section is required, other sections should be included as necessary.
exec_cmd-- path to module executable, may include command line arguments
name-- module name
description-- optional module description
has_interface-- whether the module has a web interface
key = value-- keyword arguments (these may also be specified in the
name = /stream/path-- input Pipe Configuration
name = /stream/path-- output pipe configuration
Note: Reader Modules may only have a single output and no inputs. Filter modules have no restrictions on the number of inputs and outputs.
Pipes connect modules to streams and are configured in the [Inputs] and [Outputs] section of the Module Configuration file. At a minimum the configuration specifies a pipe name and a stream path shown in Example 1 below.
The pipe configuration can also include an inline stream configuration. This can be used in place of a Stream Configuration
file or in addition to it. Using both enables static type checking for the pipeline. The inline configuration is
separated from the stream path by a colon
:. The stream datatype is followed by a list of comma separated element names
enclosed with brackets
[ ]. If
the stream is not explicitly configured or does not already exist in the database it is created with default
attributes. In Example 2 above the
inline pipe is connected to
which has three
float32 elements named
z. If this stream already exists
with a different datatype or number of elements, Joule will not start the module.
Pipes can also connect to remote streams. To specify a remote source or destination add the URL and optional port
number before the stream path. The URL is separated from the stream path by a single space. Remote pipes must include an inline stream configuration.
In example 3 above the
remote pipe is connected to
node2.net. If this stream does not
exist on node2, it will be created with default attributes. If it does exist with a different datatype, or number of
elements, Joule will not start the module.
Streams can be connected to multiple input pipes but may only be connected to a single output pipe. If a module attempts to connect an output pipe to a stream that already has a producer, Joule will not start the module.
Streams are timestamped data flows. They are composed of one or more elements as shown below. Timestamps are in Unix microseconds (elapsed time since January 1, 1970).
Timestamp Element1 Element2 ... ElementN 1003421 0.0 10.5 ... 2.3 1003423 1.0 -8.0 ... 2.3 1003429 8.0 12.5 ... 2.3 1003485 4.0 83.5 ... 2.3 ... ... ... ... ...
The configuration format is shown below:
Stream configuration files must end with the .conf suffix and should be placed in /etc/joule/stream_configs. Both [Main] and [Element1] are required. For streams with more than one element include additional sections [Element2], [Element3], etc. See the list below for information on each setting.
name-- stream identifier, white space is permitted
path-- unique identifier which follows the Unix file naming convention. The web UI visualizes the path as a folder hierarchy.
datatype-- element datatype, must be one of the following values:
float32 int8 uint8 float64 int16 uint16 int32 uint32 int64 uint64
keep-- how long to store stream data. Format is a value and unit. Units are h: hours, d: days, w: weeks, m: months, y: years. For example
6dwill keep the last six days of data. Specify
noneto keep no data or
allto keep all data.
decimate-- whether decimated data will be stored for this stream. Decimation roughly doubles the required storage but enables web UI visualization.
name-- element identifier, may contain whitespace
- NOTE: the following settings apply only to visualizations in the web UI
plottable-- [yes|no] whether the element can be plotted
type-- [continuous|discrete|event] controls the plot type
offset-- apply linear scaling y=scale_factor*(x-offset)
scale_factor-- apply linear scaling y=scale_factor*(x-offset)
default_max-- control axis scaling, leave blank to auto scale
default_min-- control axis scaling, leave blank to auto scale
Streams may also be configured using an abbreviated inline syntax in a module's Pipe Configuration.
Joule uses a set of default configurations that should work for most cases. These defaults can be customized by editing /etc/joule/main.conf. Start joule with the --config flag to use a configuration file at an alternate location. The example main.conf below shows the full set of options and their default settings:
See the list below for information on each setting.
url-- address of NilmDB server
InsertionPeriod-- how often to send stream data to NilmDB (in seconds)
CleanupPeriod-- how often to remove old data as specified by stream keep parameters
DbPath-- path to sqlite database used internally by joule
MaxLogLines-- max number of lines to keep in a module log file (automatically rolls)
ModuleDirectory-- folder with module configuration files (absolute path)
StreamDirectory-- folder with stream configuration files (absolute path)
ModulesDocs-- JSON data file for module documentation
IPAddress-- address to listen for standalone modules
Port-- port to listen for standalone modules