Serialization

KeySquare can operate with a number of serialization formats (see below). It natively uses SBE (Simple Binary Encoding).

The performance benefits of using SBE over other serialization methods such as Protocol Buffers are well documented. However, given the performance benefits, you may question why it has not gained widespread adoption outside of trading applications where performance and latency are paramount. We believe there are several reasons for this:

• Performance over developer productivity
  • Outside of financial services, performance is less critical compared to developer productivity
• Complex tooling and setup
  • Code generation using SBE Tool is clunky compared to something such as protoc
• Smaller user community compared to other serialization formats
• Schemaless formats such as JSON are inherently more readable as they are self-describing
  • Versioning and backward compatibility are harder
• Idiosyncratic behaviour
  • Streaming reads are performant but require linear access
• Schema evolution is possible but requires discipline

KeySquare has looked to address these deficiencies with:

• Developer productivity improvements
  • Data schemas are owned by components and can be evolved independently
  • Data schemas are published by components and KeySquare components like the Data Viewer do not need to be upgraded to support viewing new schemas
  • Data schema changes are simple to make
• POJOs, Maps and DTOs
  • If your application is not latency sensitive, you can prioritise flexibility over performance
• Schemaless data (encapsulated in an SBE envelope) gives developer flexibility and enables features like Excel support

The platform has support for a number of formats:

Serialization	Has Schema	Performance	Description
Raw Simple Binary Encoding	✔️	🟩🟩🟩	Simple Binary Encoding (SBE) is the primary form of serialization on the platform. It is a high-performance binary encoding protocol designed for low-latency applications. SBEs use Encoders and Decoders for writing and reading data respectively. These are simple flyweights wrapping a memory buffer that access data types within the SBE at fixed offsets for static fields at the start of the message. Repeating groups and variable length fields are positioned at the end of the message. Note, these need to iterated through in sequence.
Data Transfer Objects	✔️	🟩🟩⬜	Data Transfer Objects (DTOs) are Java types that are generated using SBE schema. These allow direct interoperability with SBE and are simpler to use than pure SBE. DTOs are suited to applications where latency is less of a concern. KeySquare recommends using DTOs for most use-cases unless absolutely performance sensitive. DTOs provide developer productivity benefits at minimal performance cost. Benefits: Safely deal with repeating group structures into data structures, avoiding potential pitfalls Avoid dealing with complexities of split SBE encoders/decoders
Flexible Messaging	❌	🟩⬜⬜	Java POJOs, Maps or other data types can be automatically serialized to JSON. Suited to rapid prototyping or applications where latency isn't a concern.
Custom	-	-	Custom serialization formats can be used by enveloping them within a SBE wrapper.

Canonical Models

KeySquare provides a canonical model that defines the business models required for Fixed Income. Applications can publish using these same canonical models which will allow:

• Interoperability with KeySquare eTrading Application Suite
• Interoperability with KeySquare Data Analytics Suite

Custom Models

Custom models can be defined using a SBE schema file. The client-model sample project shows an example of how this can be done. In particular, examine:

• pom.xml - this contains build targets to support code generation from schema.
• src/main/resources/sbe-model-client.xml - the schema file follows the following format:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe" xmlns:xi="http://www.w3.org/2001/XInclude"
  package="<package>"
  id="<id>"
  version="<version>"
  semanticVersion="<semanticVersion>"
  description="<description>"
  byteOrder="littleEndian">

  <!-- this file is auto-extracted as part of the build process and contains core platform information -->
  <xi:include href="../../../target/keysquare/sbe-model-core-definintions.xml" />

  ... type definitions ...

</sbe:messageSchema>

In the schema example above the following placeholders should be replaced as appropriate:

Tag	Description
<package>	The Java package name where generated encoders / decoders code should be placed.
<id>	A unique numerical identifier for the schema. (Note: 900 - 905 are reserved for platform use)
<version>	The version of the schema.
<semanticVersion>	The friendly schema name which will be used for display purposes.
<description>	A description of the schema.

info

When defining a custom model the first field should always be a keyHeader, which is defined in the sbe-model-core-definintions.xml provided by KeySquare.

 <sbe:message name="Message" id="1">
    <field name="keyHeader" id="1" type="keyHeader" />
    ... other fields ...
  </sbe:message>

tip

Speed up development workflow by using IDE integration: Every time you edit and save a schema file, a build target in the pom.xml will run automatically in both VS Code and Eclipse IDEs and auto generate Java model classes.

Eclipse setup click here for details.

IntelliJ setup click here for details.