Platform Best Practices

The KeySquare Platform can be run in a number of configurations depending on performance requirements. The main trade-off is hardware resource usage versus latency.

• If hardware resources are constrained, you may choose to use fewer CPU resources at the cost of higher latency.
• If hardware resources are available, best practice is to isolate CPU cores, pin important threads to CPUs and ensure applications have sufficient memory headroom.

CPU Pinning

Aeron Media Driver

The Media Driver has three important threads: Sender, Receiver and Conductor.

The Media Driver already defaults to the recommended low-latency threading mode and idle strategies. These properties do not need to be set explicitly. If they are present in a host configuration, they should ideally retain the default values:

Property	Recommended default
KS_AERON_THREADING_MODE	DEDICATED
KS_AERON_MEDIA_CONDUCTOR_IDLE_STRATEGY	org.agrona.concurrent.BusySpinIdleStrategy
KS_AERON_MEDIA_RECEIVER_IDLE_STRATEGY	org.agrona.concurrent.NoOpIdleStrategy
KS_AERON_MEDIA_SENDER_IDLE_STRATEGY	org.agrona.concurrent.NoOpIdleStrategy

Changing these defaults reduces the Media Driver's CPU requirement at the cost of latency and should be a deliberate capacity decision.

Sequencer, Relay Live and Relay Cache

The Sequencer, Relay Live and Relay Cache each have a ks thread that runs their main business logic. Their KS_APP_APPLICATION_IDLE_STRATEGY property defaults to org.agrona.concurrent.NoOpIdleStrategy and does not need to be set explicitly. If it is present in a host configuration, it should ideally retain that default value.

Core Isolation and Pinning

Use TuneD's cpu-partitioning profile to isolate CPUs for latency-sensitive work. Start by displaying the host's CPU layout:

lscpu -e=CPU,NODE,SOCKET,CORE,ONLINE

Interpret the columns as follows:

Column	Meaning
CPU	The logical CPU ID used in TuneD and KeySquare configuration.
NODE	The NUMA node containing the CPU. Prefer CPUs from one node where capacity permits.
SOCKET and CORE	Together identify a physical core. Rows with the same socket and core are SMT siblings.
ONLINE	Whether the logical CPU is available.

Before selecting a NUMA node, identify the interface that carries KeySquare traffic and determine the NIC's NUMA node:

cat /sys/class/net/<data-interface>/device/numa_node

If the command reports a NUMA node such as 0, prefer CPU rows with the same NODE value. A value of -1 means that the kernel does not expose a specific NUMA node for the device.

The baseline platform requires six exclusive physical cores for its latency-sensitive threads:

Component	Exclusive cores
Aeron Media Driver	3: Sender, Receiver and Conductor
Sequencer	1
Relay Live	1
Relay Cache	1

Ideally, add one exclusive core for each KeySquare business application, e.g. Static Data Server.

After calculating the total, select that number of different physical cores, preferably from the NUMA node local to the data NIC. Use one logical CPU from each selected core for a latency-sensitive thread. When a physical core has a second SMT row, add that sibling to isolated_cores as well but leave it unused. If every socket/core combination has only one row, SMT is disabled and there is no sibling to add. Leave CPU 0 and enough complete physical cores outside the isolated set for operating-system, JVM support and NIC interrupt work.

Configure the host-specific CPU lists in /etc/tuned/cpu-partitioning-variables.conf:

isolated_cores=<isolated-cpu-list>
no_balance_cores=<isolated-cpu-list>

For example, one KeySquare canary server uses:

# KeySquare canary: CPUs 8-15 pinned; CPUs 24-31 are idle SMT siblings.
isolated_cores=8-15,24-31
no_balance_cores=8-15,24-31

Activate the profile and reboot:

sudo tuned-adm profile cpu-partitioning
sudo reboot

After rebooting, confirm that the profile is active:

tuned-adm active
tuned-adm verify

Good output identifies cpu-partitioning as the active profile and reports that verification succeeded.

Check the kernel boot options applied by TuneD:

cat /proc/cmdline
cat /sys/devices/system/cpu/isolated
cat /sys/devices/system/cpu/nohz_full

Good output includes isolcpus, nohz_full and rcu_nocbs covering the configured isolated CPUs. rcu_nocbs moves RCU callbacks away from those CPUs.

KeySquare Bare Metal Configuration

When using a bare metal deployment, set the pinned affinity in each host-specific YAML process definition:

systemProperties:
  APP_CPUS: '<pinned-cpu-list>'

APP_CPUS supplies the CPUs used by KeySquare applications and must contain comma-separated CPU IDs, not ranges.

KeySquare Container Configuration

When using a container deployment, set the pinned CPU lists in conf/<hostname>.properties:

AERON_MEDIA_DRIVER_PINNED_CORES=<cpu-id>,<cpu-id>,<cpu-id>
SEQUENCER_PINNED_CORES=<cpu-id>
RELAY_LIVE_PINNED_CORES=<cpu-id>
RELAY_CACHE_PINNED_CORES=<cpu-id>

These variables will be supplied to the appropriate container which maps *_PINNED_CORES to the APP_CPUS JVM property.

Host Configuration

Configure the required host kernel settings before setting KeySquare application socket properties. Add the applicable settings together in a single site-managed .conf file under /etc/sysctl.d/:

# UDP socket ceilings
net.core.rmem_max=16777216
net.core.wmem_max=16777216

# TCP buffers: minimum, initial and maximum
net.ipv4.tcp_rmem=4096 2097152 16777216
net.ipv4.tcp_wmem=4096 2097152 16777216

# Disable automatic page migration when CPU and memory placement is deliberate
kernel.numa_balancing=0

# Allow allocations from another NUMA node instead of reclaiming local memory
vm.zone_reclaim_mode=0

# Minimise swapping when the complete workload fits safely in RAM
vm.swappiness=0

Apply all selected settings before starting the KeySquare Platform:

sudo sysctl --system

KeySquare Network Configuration

MTU and Socket Buffers

A standard network MTU of 1500 bytes is assumed with an Aeron MTU setting of 1408 bytes applied to account for network headers. On a verified end-to-end network with an MTU of at least 9000 bytes, use an Aeron MTU of 8192 bytes. Do not set the Aeron value equal to the network MTU because space is required for network headers.

With the host socket ceilings applied, configure the Media Driver to use 2 MiB Aeron buffers:

Bare Metal

Add the settings to the Media Driver's host-specific process definition:

systemProperties:
  aeron.rcv.initial.window.length: '2097152'
  aeron.socket.so_rcvbuf: '2097152'
  aeron.socket.so_sndbuf: '2097152'
  # only for a verified network MTU of at least 9000 bytes:
  # aeron.mtu.length: '8192'

Container

Add the equivalent environment variables to the aeron-media-driver service in the Compose YAML:

services:
  aeron-media-driver:
    environment:
      VMARG_aeron.rcv.initial.window.length: '2097152'
      VMARG_aeron.socket.so_rcvbuf: '2097152'
      VMARG_aeron.socket.so_sndbuf: '2097152'
      # only for a verified network MTU of at least 9000 bytes:
      # VMARG_aeron.mtu.length: '8192'

The initial receive window must not be greater than the receive socket buffer. After starting the Media Driver, record the UDP error counters before a representative peak-load test:

nstat -az UdpInErrors UdpRcvbufErrors UdpSndbufErrors

Run the same nstat command after the test. The Media Driver should start without socket-buffer warnings, and UdpRcvbufErrors and UdpSndbufErrors should not increase. If either counter increases, investigate CPU starvation, receiver capacity and the network path before testing a larger Aeron buffer. Raising only the operating-system ceiling cannot resolve a full socket buffer.

Operating System Services

List the running services:

systemctl list-units --type=service --state=running --no-pager

Review each service and identify any unnecessary periodic CPU, memory, storage or network work.

Check for failed services:

systemctl --failed --no-pager

Good output reports no failed units. Resolve unexpected failures before using the host.

Finally, inspect all process threads and the CPUs on which they most recently ran:

ps -eL -ww -o user,pid,tid,psr,pcpu,comm,args --sort=-pcpu

The ps output has one row per thread. PID identifies the owning process, TID identifies the thread, PSR shows the logical CPU on which it most recently ran, COMMAND shows the thread name, and the final column contains the full process command line. For the Media Driver, good output shows the Sender, Receiver and Conductor each using its assigned CPU.

Identify services that are not required for KeySquare or operation of the host and that perform periodic CPU, memory, storage or network work. There is no universal list: monitoring, security scanning, backups and update services may be mandatory, but their intensive work should be scheduled outside latency-sensitive periods where possible. Do not disable TuneD, networking, remote access, clock synchronisation or services required by the host's operational and security policies.

Further Optimisation

The settings above provide the recommended baseline. The following options can improve latency in some environments, but should not be applied without a representative load test, before-and-after tail-latency measurements and an operational review. Change one area at a time and do not copy values from another host without sizing them for the target system.

Area	Option to investigate	Potential benefit and required checks
NUMA locality	Bind process memory to the NUMA node containing the pinned CPUs and data NIC, while keeping JVM and operating-system support work on non-isolated CPUs.	Can reduce remote-memory latency on multi-node hosts. Strict binding can exhaust one node while another still has free memory, so validate node-level headroom, JVM behaviour and the deployment mechanism. It provides no benefit on a single-node host.
Virtual-memory accounting	Increase vm.stat_interval.	May reduce periodic kernel accounting activity, but the expected improvement is small and memory statistics become less current. Consider it only after the primary CPU isolation and memory-headroom work is complete.
Kernel memory reserve	Increase vm.min_free_kbytes.	A larger reserve can reduce direct-reclaim stalls, but also reduces memory available to applications and can cause earlier out-of-memory conditions. Retain the operating-system default unless measurements show reclaim stalls, then derive and test a host-specific value.
Kernel UDP receive path	Increase net.core.netdev_max_backlog or enable net.core.busy_poll and net.core.busy_read.	A larger backlog may absorb transient bursts but can add queueing and mask receiver starvation. Busy polling can reduce wake-up latency but consumes additional CPU and power and depends on kernel, NIC and socket support. Test these separately and retain them only when drop counters and tail latency improve.
Storage writeback	Adjust vm.dirty_background_ratio, vm.dirty_ratio and vm.dirty_expire_centisecs, or review filesystem access-time options such as noatime.	Can move filesystem work away from the latency-sensitive path, but may instead create larger writeback bursts or increase the amount of unwritten data at risk. Test with the actual storage, logging and durability requirements; do not use options that disable write barriers without a separate durability assessment.
TCP connection handling	Increase net.ipv4.tcp_max_syn_backlog, reduce net.ipv4.tcp_syn_retries or enable net.ipv4.tcp_tw_reuse.	KeySquare Proxy normally uses long-lived connections, so these settings do not improve established-stream latency. Consider them only when measurements show connection-backlog overflow, slow failed connection attempts or high short-lived connection churn. They alter connection recovery and reuse behaviour; tcp_tw_reuse in particular should only be changed after expert network review.
Profiling access	Relax kernel.perf_event_paranoid, kernel.kptr_restrict or related profiling limits when kernel-level profiling is required.	Enables more complete perf and async-profiler output but does not improve runtime performance. It exposes additional kernel information, so use values approved by the client's security policy and restore restrictive settings when profiling is complete.