SBE and DTOs

This page shows how to generate SBE encoders, decoders and DTOs from a SBE schema.

For more information on serialization, see Serialization.

Maven Setup

Add and adapt the following into your project. This will:

• Generate SBE encoders, decoders, DTOs and SBE IR files.
• Add the generated sources and resources to the project.

info

The example below uses sbe-sample-model.xml as the schema file name. This can be replaced with a project specific model name.

<build>
  <plugins>

    <!-- step 1: generate SBE encoders, decoders and DTOs -->
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>exec-maven-plugin</artifactId>
      <version>${exec-maven-plugin.version}</version>
      <executions>
        <?m2e execute onConfiguration,onIncremental?>
        <execution>
            <id>generate-sbes-and-pojos</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>exec</goal>
            </goals>
            <configuration>
                <executable>java</executable>
                <arguments>
                    <argument>--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED</argument>
                    <argument>-Dsbe.output.dir=${project.build.directory}/generated/src/main/java</argument>
                    <argument>-Dsbeir.output.dir=${project.build.directory}/generated/src/main/resources</argument>
                    <argument>-cp</argument>
                    <classpath/>
                    <argument>trading.keysquare.codegen.Main</argument>
                    <argument>${project.basedir}/src/main/resources/sbe-sample-model.xml</argument>
                </arguments>
            </configuration>
        </execution>
      </executions>
      <configuration>
        <includeProjectDependencies>true</includeProjectDependencies>
        <includePluginDependencies>true</includePluginDependencies>
        <workingDirectory>${project.build.directory}/generated/src/main/java</workingDirectory>
      </configuration>
    </plugin>

    <!-- step 2: add the generated sources and resources to the build path -->
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>build-helper-maven-plugin</artifactId>
      <version>${build-helper-maven-plugin.version}</version>
      <executions>
        <execution>
          <?m2e execute onConfiguration,onIncremental?>
          <id>add-source</id>
          <phase>generate-sources</phase>
          <goals>
            <goal>add-source</goal>
          </goals>
          <configuration>
            <sources>
              <source>
                ${project.build.directory}/generated/src/main/java/</source>
            </sources>
          </configuration>
        </execution>
        <execution>
          <id>add-resource</id>
          <phase>generate-sources</phase>
          <goals>
            <goal>add-resource</goal>
          </goals>
          <configuration>
            <resources>
              <resource>
                <directory>${project.build.directory}/generated/src/main/resources/</directory>
              </resource>
              <resource>
                <directory>${project.build.directory}/keysquare/</directory>
              </resource>
            </resources>
          </configuration>
        </execution>
      </executions>
    </plugin>

  </plugins>
</build>

Example SBE Schema

The SBE Schema should be placed in the resources directory. Based on the Maven plugin example above, this would be:

<project-root>/src/main/resources/sbe-sample-model.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe" xmlns:xi="http://www.w3.org/2001/XInclude"
  package="trading.keysquare.model.sample"
  id="1"
  version="1"
  semanticVersion="Sample Model"
  description="Sample SBE Models"
  byteOrder="littleEndian">

  <types>

    <!-- Standard SBE type definitions for: messageHeader, varStringEncoding, groupSizeEncoding and varDataEncoding-->
    <composite name="messageHeader" description="Message identifiers and length of message root">
      <type name="blockLength" primitiveType="uint16" />
      <type name="templateId" primitiveType="uint16" />
      <type name="schemaId" primitiveType="uint16" />
      <type name="version" primitiveType="uint16" />
    </composite>

    <composite name="varStringEncoding">
      <type name="length" primitiveType="uint32" maxValue="1073741824" />
      <type name="varData" primitiveType="uint8" length="0" characterEncoding="UTF-8" />
    </composite>

    <composite name="groupSizeEncoding" description="Repeating group dimensions.">
      <type name="blockLength" primitiveType="uint16" />
      <type name="numInGroup" primitiveType="uint16" />
    </composite>

    <composite name="varDataEncoding">
      <type name="length" primitiveType="uint32" maxValue="1073741824" />
      <type name="varData" primitiveType="uint8" length="0" />
    </composite>

    <!-- Sample type definitions-->
    <type name="char10" primitiveType="char" length="10" />
    <type name="char100" primitiveType="char" length="100" />

  </types>

  <sbe:message name="SampleMessage" id="5000" description="Sample Message">
    <field name="id" id="1" type="char10" semanticType="id" />
    <field name="message" id="2" type="char100" />
    <field name="price" id="3" type="double" />
    <field name="qty" id="4" type="int64" />
  </sbe:message>

</sbe:messageSchema>

Using Generated Classes

The schema above generates SampleMessageEncoder, SampleMessageDecoder and SampleMessageDto.

SBE messages can be routed and subscribed using the generated decoder:

session.routing().sbe(SampleMessageDecoder.class, this::processSampleMessage, Optional.empty());

Topic sbeTopic = Topics.create(SampleMessageDecoder.class, Topic.SOURCE_WILDCARD, "samples", "sample-1");
session.subscribe(SubscriptionType.LIVE, count -> {}, sbeTopic);

private void processSampleMessage(KeyMessage<SampleMessageDecoder> message) {
  SampleMessageDecoder sample = message.payload();
  log.info("Received {}", sample);
}

SBE messages can be published using the generated encoder.

SbeMessage<SampleMessageEncoder, SampleMessageDecoder> message =
    Messages.sbe(SampleMessageEncoder.class);

SampleMessageEncoder encoder = new SampleMessageEncoder();
encoder.wrapAndApplyHeader(new ExpandableArrayBuffer(), 0, new MessageHeaderEncoder());
encoder.id("id-1");
encoder.message("Sample message");
encoder.price(99.125);
encoder.qty(10_000_000);

message.topic(session.getApplicationName(), "samples", "sample-1");
message.payloadEncoder(encoder);

Result result = session.publish(message);

The DTO generated from the same schema can be routed and subscribed using the DTO APIs:

session.routing().dto(SampleMessageDto.class, this::processSampleMessageDto, Optional.empty());

Topic dtoTopic = Topics.create(SampleMessageDto.class, Topic.SOURCE_WILDCARD, "samples", "sample-1");
session.subscribe(SubscriptionType.LIVE, count -> log.info("Subscription complete, received {}", count), dtoTopic);

private void processSampleMessageDto(KeyMessage<SampleMessageDto> message) {
  SampleMessageDto sample = message.payload();
  log.info("Received {}", sample);
}

DTO messages can be published using the generated DTO:

SampleMessageDto sampleDto = new SampleMessageDto();
sampleDto.id("id-1");
sampleDto.message("Sample message");
sampleDto.price(99.125);
sampleDto.qty(10_000_000);

DtoMessage<SampleMessageDto> dtoMessage = Messages.dto(SampleMessageDto.class);
dtoMessage.topic(session.getApplicationName(), "samples", "sample-1");
dtoMessage.payload(sampleDto);
session.publish(dtoMessage);

info

DTOs and SBE messages use the same schema-backed wire format. Whether an application publishes using the generated SBE encoder or the generated DTO, the message is carried on the platform in SBE form.

Each subscribing application can then choose the representation that suits it best. A low-latency component can receive the message as SBE, while another application can receive the same message as a DTO and work with normal Java objects.