Using raw TCP

Hivekit comes with SDKs and a well structured REST API that makes it easy to integrate with our platform. But, there are cases where it makes sense to use Hivekit's protocol directly via TCP. Usually, when you want to sent data from a low powered Vehicle or IoT device.

Concepts

Before we start, let's quickly review the two concepts we'll be working with.

A realm is a space within which something happens, e.g. a city or a building. Realms serve as containers for all other concepts - your vehicles, areas, instructions and tasks all live within the scope of a realm.
An object is a physical or virtual entity that can be tracked and managed. This could be a vehicle, a person, a sensor, a package, a building, a piece of equipment, or anything else you want to keep track of. Objects have a location and optional data.

Protocol

Hivekit uses a custom, JSON based message format to communicate via WebSockets, TCP and other connection types. Each message consists of an array of "actions". An action relates to a certain realm and type (e.g. "object", "area" etc) and a specific action (e.g. "create", "list", "read", "update" etc). Here's what that looks like:

[{// create an object
  typ: 'obj', // type == object
  act: 'cre', // action == create
  id: 'rider/SadeIbrahim', // object id
  rea: 'lagos', // realm id
  loc: { lon: 3.37590, lat: 6.51309 } // location
}, {
    // read that same object
    typ: 'obj', // type == object
    act: 'rea', // action == rea
    id: 'rider/SadeIbrahim', // object id
    rea: 'lagos', // realm id
}]

As you can see, the fieldnames and some of the field values are abbreviated as three letter short codes. This helps to keep the message size small and the protocol efficient. You can find a full list of these codes here.

What we want to do

In this tutorial, we'll

sign in to Hivekit
connect to the Hivekit TCP endpoint
authenticate the TCP connection as soon as its open
create a new object
check that object on a map
subscribe to updates
update the object on the map and see the updates come in via the subscription

Start by logging into your Hivekit account. If you don't have an account yet, you can sign up for free at https://hivekit.io/signup/.

Dashboard after signup

When you first sign up, Hivekit will automatically create an initial realm and an open access token.

Step 2: Find the message structure for an Object creation

We maintain a list of actions and related responses in our protocol docs. Here you can find the documentation for an object create action.

Step 3: Gather the necessary information

We need to gather the following information:

The API Endpoint URL to connect to
The Realm ID
The Access Token
The Object ID

The endpoint IP to connect to

You can connect to the TCP endpoint at 54.170.19.61:89. This is a static IP for a server cluster based in Ireland. We do however provide globally distributed endpoints and you will get much better performance if you use the endpoint closest to your location. To get the best suited endpoint for your location, we recommend using the DNS name api.hivekit.io and resolving it to an IP address before connecting, e.g. via the dig command or a similar language feature.

The Realm ID

The realm ID can be found on the Hivekit dashboard, just below the selected realm:

The Access Token

The access token can be found in the "Access Management" section of the Hivekit dashboard.

You can display it and copy its value by clicking "show token". Access token

The Object ID

The object id is something we need to generate when creating the object. You can either use an existing id, e.g. a license plate number for a vehicle or come up with your own. Object IDs allow you to identify an object throughout its lifecycle. For this tutorial, we'll create a randon id, using the nanoid library.

Step 4: Connect and Authenticate

We'll be using NodeJS and the built in net module for this example, but it works just the same with any other language or TCP implementation. Once your connection is established, the very first message needs to be Bearer <YOUR-TOKEN>\n. If you send anything else, the server will close the connection.

import net from 'net';
import { nanoid } from 'nanoid';

const ip = '54.170.19.61';
const port = 89;
const token = 'YOUR_TOKEN';
const realmId = 'YOUR_REALM_ID';
const objId = 'Object-' + nanoid();
const client = new net.Socket();

client.on('error', console.error);

client.connect(port, ip, function () {
	client.write('Bearer ' + token + '\n');
});


client.on('data', function (rawData) {
    console.log('Received: ', data);
});

If you run this code, you should receive the following message from the server:

[
  {
    typ: 'sys', // type = system
    act: 'aut', // action = authentication
    res: 'suc', // result = success
    dat: { // details about the server you're connected to
        version: '1.3.2', 
        buildDate: 'Fri Feb 16 17:15:33 UTC 2024' }
  }
]

Step 5: Process incoming messages

Once we've received this message, we know that our connection was succesfully authenticated and that we can start sending our own messages. So, let's extend our 'data' callback to parse the incoming messages and act on them.

client.on('data', function (data) {
    // data is a JSON array of objects, each object is a message from the server
    const messages = JSON.parse(data);

    messages.forEach(msg => {
        // If this message is a System Authentication Success message, we can start sending our own messages
        if (msg.typ === 'sys' && msg.act === 'aut' && msg.res === 'suc') {
            init();
        // If we don't have a registered callback for this message, we'll log it
        } else {
            console.log('Unhandled message', msg);
        }
    })
});

Step 6: Request - Response Communication

Most actions, such as creating, updating, listing, reading or deleting require request-response communication. To facilitate that over TCP, Hivekit uses a correlation id. This is a random string that we generate on the client and send with our request. The server will then include this correlation id in its response, so we can map the response to the request. Here's how we can implement this in our code:

const callbacks = {};

// Send a message to the server
function send(msg, callback) {
    // We'll create a random correlation id to map the response to the request
    const correlationId = nanoid(); 
    // We'll add the correlation id to the message
    msg.cid = correlationId;
    // We'll store the callback to invoke it when we receive a message with this correlation id
    callbacks[correlationId] = callback;
    // We'll send the message to the server. Notice the array around the message, this is the format the server expects
	client.write(JSON.stringify([msg]) + '\n');
}

// A wrapper function that let's us use send as an async function
function sendAndAwaitResponse(msg) {
    return new Promise((resolve, reject) => {
        send(msg, resolve);
    });
}

We then extend our on-message callback to invoke the registered callback when we receive a message with the correlation id:

//...
// If we have a registered callback for this message, we'll invoke it
else if (msg.cid && callbacks[msg.cid]) {
    callbacks[msg.cid](msg);
    delete callbacks[msg.cid];
}

Step 7: Create an object

Phew - that was a lot of setup. But now, it's finally time to create our object. We'll create the init function that our code calls as soon as the connection is authenticated and use the sendAndAwaitResponse function we just created to send a message to the server and wait for the response. Here's how we can do that:

async function init() {
    // Create an object
    let result = await sendAndAwaitResponse({
        typ: 'obj', // type = object
        act: 'cre', // action = create
        id: objId, // our object id
        rea: realmId, // our realm id
        lab: 'Object A', // label
        loc: { lon: 13.404954, lat: 52.520008 }, // location
        dat: { type: 'scooter', charge: 0.5 }, // data
    });
    console.log('Create Object: ', result);
}

The server will respond to that with a message like

[
  {
    id: 'object-DpV4jVAmcGDkb2cQr73YR',
    cid: 'AUeaiQQYZEQgwT-H1l0s4',
    res: 'suc'
  }
]

which will be processed by our message function and forwarded to the correct callback.

Step 8: Check the object on a map

Let's check if our object really exists. We can do this by visiting the Realms section of the Hivekit dashboard and clicking on "Go to Realm Map". Here, we should find our new object on the category tree on the left - as well as on the map itself. When we click on the object, we should see its label, location and data.

Map view

Step 9: Read the object

Reading the object is almost the same as creating it. We simple change the action in our request to 'rea' and remove the lab, loc and dat fields. Here's what that looks like:

 // Read the object
result = await sendAndAwaitResponse({
    typ: 'obj',
    act: 'rea',
    id: objId,
    rea: realmId,
});
console.log('Read Object: ', result);

The resulting output should be:

[{
  id: 'object-DpV4jVAmcGDkb2cQr73YR', // object id
  cid: 'wLX7LuzLa1JnVvhKinW2R', // correlation id
  res: 'suc', // result = success
  dat: { // response data
    cst: 'dis', // connection state, see https://hivekit.io/guides/core/presence
    dat: { charge: 0.5, type: 'scooter' }, // data
    lab: 'Object A', // label
    loc: {
      acc: 0, // accuracy
      alc: 0, // altitude accuracy
      alt: 0, // altitude
      gcs: '', // geospatial coordinate system (empty for lat/lon)
      hea: 0, // heading
      lat: 52.520008, // latitude
      lon: 13.404954, // longitude
      spe: 0, // speed in m/s
      tim: '2024-02-20T11:01:13.816689183Z' // time this location was recorded in UTC
    }
  }
}]

Step 10: Subscribing to updates

Finally, we want to get a realtime feed of any change to any object within our realm. For that, we need to create a subscription. Hivekit's subscriptions have a lot of options and settings and let you e.g. subscribe to areas, instructions, a single object and its tasks, objects within a certain area or within a radius around another moving object or objects with certain criteria, e.g. a charge below 20%. But for now, let's keep things simple and just subscribe to all objects within our realm.

We create a subscription by sending the following message:

// Create a subscription
result = await sendAndAwaitResponse({
    typ: 'sub', // type = subscription
    act: 'cre', // action = create
    id: subscriptionId, // our subscription id
    rea: realmId, // our realm id
    dat: { // subscription data
        sty: 'rea', // scope type = realm - we want to subscribe to all objects within our realm
        sid: realmId, // scope id - the id of whatever the scope type is, in this case, the realm id
        typ: 'obj' // type = object - the type of entity we want to subscribe to
    }
});

To test our subscription, we head back to our realm map and move the object around a bit. As soon as we hit save our subscription should receive the following kind of message:

{
  "typ": "sub", // type = subscription
  "id": "sub-PxIHYszBfDDooN5FHpw-e", // subscription id
  "uty": "dta", // update type = delta. This could also be "ful" for full updates
  "seq": 1, // sequence number, should increment with each update
  "obj": { // objects that changed
    "cre": { // created means, that you need to add these object to your local state of objects
      "object-zdC1zrUv0sK3tqL-Kefod": {
        "lab": "Object A",
        "loc": {/**...**/},
        "dat": {/**...**/},
        "cst": "dis"
      }
    }
  }
}

To understand this message, it's important to understand the way subscriptions work in Hivekit. Subscriptions assume that you keep a local copy of the state of your subscription and Hivekit provides you with updates containing the changes to that subscription. So each subscription message will contain lists of objects to be added to your local state (cre), updated in your local state (upd) and removed from your local state (del). This way, you can keep your local state in sync with the server's state.

If you add "exe": true (executeImmediatly = true) to your subscription message, Hivekit will start by sending you the full state of the subscription when you create it, expressed as a message with "uty":"ful" (update type = full). This is useful if you're just starting your client and need to know the current state of the world.

Step 11: Handle subscription messages.

Each incoming message contains the subscription id it belongs to. To route updates, it makes sense to store another callback upon subscribing and to route messages to it based on the subscription id. Here's how we can do that:

// We'll create a random id for our subscription
const subscriptionId = 'sub-' + nanoid();
const subscriptionCallbacks = {};

// Register a callback for incoming subscription updates upon subscribing
subscriptionCallbacks[subscriptionId] = update => {
    console.log('Subscription Update: ', JSON.stringify(update, null, 2));
}

// Add this to the message function
// If this message is a subscription update, we'll invoke the registered callback
} else if (msg.typ === 'sub' && subscriptionCallbacks[msg.id]) {
    subscriptionCallbacks[msg.id](msg);
}

That's it.

Phew - that was complicated. Thanks for sticking with me to the end. If you want to go deeper on this, I can recommend reading up on the protocol and to look at Hivekit's JavaScript SDK as a reference implementation.

If you have any questions, feel free to ask in the Hivekit Discord Chat or Contact us via email..

Here's the full code for this example:

import net from 'net';
import { nanoid } from 'nanoid';

const client = new net.Socket();
const ip = '54.170.19.61';
const port = 89;
const token = 'YOUR_TOKEN';
const realmId = 'YOUR_REALM_ID';
const callbacks = {};
const subscriptionCallbacks = {};
const objId = 'object-' + nanoid();

client.on('error', console.error);

client.connect(port, ip, function () {
	client.write('Bearer ' + token + '\n');
});

client.on('data', function (rawData) {
	const messages = JSON.parse(rawData);

	messages.forEach(msg => {
		if (msg.typ === 'sys' && msg.act === 'aut' && msg.res === 'suc') {
			init();
		} else if (msg.cid && callbacks[msg.cid]) {
			callbacks[msg.cid](msg);
		} else if (msg.typ === 'sub' && subscriptionCallbacks[msg.id]) {
			subscriptionCallbacks[msg.id](msg);
		} else {
			console.log('Unhandled message', msg);
		}
	})
});

client.on('close', function () {
	console.log('Connection closed');
});


function send(msg, callback) {
	const correlationId = nanoid();
	msg.cid = correlationId;
	callbacks[correlationId] = callback;
	client.write(JSON.stringify([msg]) + '\n');
}

function sendAndAwaitResponse(msg) {
	return new Promise((resolve, reject) => {
		send(msg, resolve);
	});
}

async function init() {
	let result = await sendAndAwaitResponse({
		typ: 'obj',
		act: 'cre',
		id: objId,
		rea: realmId,
		lab: 'Object A',
		loc: { lon: 13.404954, lat: 52.520008 },
		dat: { type: 'scooter', charge: 0.5 },
	});
	console.log('Create Object: ', result)

	result = await sendAndAwaitResponse({
		typ: 'obj',
		act: 'rea',
		id: objId,
		rea: realmId,
	});
	console.log('Read Object: ', result)

	const subscriptionId = 'sub-' + nanoid();
	subscriptionCallbacks[subscriptionId] = update => {
		console.log('Subscription Update: ', JSON.stringify(update, null, 2));
	}

	result = await sendAndAwaitResponse({
		typ: 'sub',
		act: 'cre',
		id: subscriptionId,
		rea: realmId,
		dat: {
			sid: realmId,
			sty: 'rea',
			typ: 'obj'
		}
	});
	console.log('Create Subscription: ', result)

	result = await sendAndAwaitResponse({
		typ: 'obj',
		act: 'set',
		id: objId,
		rea: realmId,

		loc: { lon: 13.404954, lat: 52.520008 },
		dat: { type: 'scooter', charge: 0.4 },
	});
}

Concepts​

Protocol​

What we want to do​

Step 1: Sign in to Hivekit​

Step 2: Find the message structure for an Object creation​

Step 3: Gather the necessary information​

The endpoint IP to connect to​

The Realm ID​

The Access Token​

The Object ID​

Step 4: Connect and Authenticate​

Step 5: Process incoming messages​

Step 6: Request - Response Communication​

Step 7: Create an object​

Step 8: Check the object on a map​

Step 9: Read the object​

Step 10: Subscribing to updates​

Step 11: Handle subscription messages.​

That's it.​