The Kinetic Protocol (KP) is a network protocol used in a data storage system to transfer data from one host to another host over a TCP-based network, such as the Internet. The data storage system is based on a client-server architecture where Kinetic clients in general send Put and Get requests to the Kinetic server (usually a Kinetic Disk Device) to store and retrieve data respectively. The data sent and received is the form of a Key/Value pairs, where the Key serves as a persistent identifier for a given Value. The Kinetic Protocol defines a range of operation that can be seen in the Kinetic_API.

Kinetic Protocol Specification

The Kinetic protocol currently does not have a formalized specification in terms of an RFC or other documents. The specification as it sits today is reflected in the kinetic.proto file that specifically defines bits the on the wire, a sample server implementation and a set of client API libraries. So in fact, the code sits as the current specification. The Kinetic Open Storage Group will be producing a formal specification over time.

Protocol Details

Kinetic Client applications can communicate with a Kinetic Device by sending messages over a network using TCP. Each individual message is called a “Kinetic Protocol Data Unit” (Kinetic PDU) and represents an individual request or response. For example, a Kinetic Client may send a message requesting the value associated with a particular key to a Kinetic Device. The device would respond with a message containing the value. The Kinetic Protocol supports restricting the operations a requester (identity) can perform via Access Control Lists (ACLs). The protocol definition is provided in the Kinetic-Protocol GitHub Repository.

Protocol Data Unit Structure

A Kinetic Protocol Data Unit is composed of a Protocol Buffer message, containing operation metadata & key-value metadata, and the value. It is important to note that the value is not encoded in the message; it is a separate top-level component of the Kinetic PDU. Specifically, a Kinetic PDU is structured as follows:

OFFSET	TYPE	LENGTH	DESCRIPTION
0	Byte	1 Byte	Version prefix: currently the character ‘F’, denoting the beginning of the message. (The character ‘F’, the Hex value 46).
1	4 Byte Big Endian Integer	4 Bytes	The number of bytes in the protobuf message (the maximum length for protobuf messages is 1024*1024 bytes).
5	4 Byte Big Endian Integer	4 Bytes	The number of bytes in the value (the maximum length for values is 1024*1024 bytes).
9	Bytes	<= 1024*1024 Bytes	The protobuf message.
9 + Length of protobuf Message	Bytes	<= 1024*1024 Bytes	The value.

Protobuf Structure

Within a Kinetic PDU, the message encodes the specifics of the requested operation (or response). At a high level, each message contains:

A single command
An HMAC of the byte representation of the command
Each command contains a:

Header, containing metadata about the message such as type (e.g. GET, GET_RESPONSE, PUT, PUT_RESPONSE, etc)
Body, containing operation-specific information, such as key-value information for PUT or key range information for GETKEYRANGE.
Status, containing information about whether an associated operation succeeded or failed (and why).