Javascript multiplayer concepts: cutting off unchanged data

Here is a concept which I use on my snapshot based multiplayer games to reduce ammount of sent data. The idea is basically to send only the data which have changed since last snapshot.

Let’s assume we have array of objects representing some ships. I recommend klass library for object-oriented javascript (omg, I really said that – I recommend vanilla js now), but for simplicity sake we gonna do it in raw prototypal approach (that’s better). Before we start make sure that you are familiar with extend function which can be found in libraries like jQuery or underscore.

We have to monitor which data has changed. As javascript doesn’t have convinient and fast method to overload operators (getters, setters) we are left with two resonable options:

  • defining set/get (key, value) function and use it instead of operator =
  • keep an array of last sent values for further comparison

Second option will be in charge this time.

function Ship(args) {
  // id corresponding with index of this element in ships collection array/object
  $.extend(this, {
    id: 0,
    x: 0,
    y: 0,
    direction: 0,
    destination: [0, 0],
    type: "fighter",
    foo: "something",
    bar: "i like trains",
    // values from our last snapshot
    lastExportedValues: {}
  }, args);
}

Properties foo and bar represents variables which are usable only for server and we shouldn’t waste bandwidth for sending them.

// keys for export
Ship.prototype.exports = ["x", "y", "direction", "destination", "type"];

Now the export function which will give us an object containing only properties which have changed. Also we want to get rid of key names from our JSON so we have to inform receiver which keys/indexes has been skipped. We simply create a string like “1000100″ where 0 means that key responding it’s position is not included in packet.

// ships collection
var ships = {};

Ship.prototype.export = function() {
  // packet as array, we don't need {keys: values}
  var result = [];
  // which values has been exported/changed ?
  var exported = ""; 

  for(var i = 0; i < this.exports.length; i++) {
    // for readibility
    var key = this.exports[i];
    // if value has changed since last export push it into results
    if(this.lastExportedValues[key] != this[key]) {
      this.lastExportedValues[key] = this[key];
      result.push(this[exports[i]]);
      // set flag which tells that we sent this key or no
      exported += "1";
    } else exported += "0";
  }
  // tell receiver which ship is this
  result.unshift(this.id);
  // tell receiver what have we exported
  result.unshift(exported);

  return result;
}

So our single ship now looks like [id, exportedValues, data1, data..x] example: [3, "100010", 3, "fighter"...]

function synchronize() {
  // this array will contain our "compressed" packet
  var packet = [];
  // we use associative array so we have to waste some ops for var in loop
  for(var i in ships) {
    packet.push(ships[i].export());
  }

  // send with your favorite mean of transport, ex. Websockets or AJAX
  send(packet);
}

Now for the client/receiver:

function onMessage(data) {
  // I assume you have had already decoded data to object and it contains an array of packed ships
  for (var i = 0, len = data.length; i < len; i++) {

    var packedShip = data[i];

    // remember that our ship is still a numeric array so we have to translate it to usable state
    var ship = {}

    // as we remember first key was ship's ID
    var id = packedShip.shift();
    // and the second will give us info which values has been exported
    var exported = packedShip.shift();

    // at which value in exported packet are wee looking now
    var lookupIndex = 0;

    // let's loop through values which could be exported
    for (var j = 0; j < Ship.prototype.exports.length; j++) {
      var key = Ship.prototype.exports[j];
      // if the key was exported let's take the value under current lookup index and move it one place right
      if (exported[key] == "1") {
        ship[key] = data[lookupIndex++];
      }
      // if value hasn't been exported we just do nothing and lookup cursor stands its ground
    }

    console.log(ship);

    // hooray now ship contains our exported object, we have ID so we can
    // for example insert this object into collection if it doesn't exist
    // or update if it does
  }
}

Need more compression ? Tips for minimizing bandwidth usage in HTML5 + websockets games.

How to send binary data between node.js and browser using websockets

This article is over complicated. If you are not sending binary data (streams, files) – YES use BiSON – but send it as a normal UTF-8 without encoding it to uInt Array

tested only on chrome 16.0.912.75, will check more browsers soon

Subject:

Find more efficient way to exchange JSON data between server and client in multiplayer game based on node.js and websockets.

Background:

After jugling with LZW, huffman’s algorithm and some other string compression concepts I eventually found an idea about replacing text JSON with more binary approach. In my research I managed to decrease bandwidth usage from:

14 KB/s to 11KB/s using LZW
14 KB/s to 9KB/s using BiSON (which I will talk about)
14 KB/s to 6KB/s using BiSON + fixed size packets

Solution:

You will need:

  • Ivo Wetzel’s BiSON library
  • Node.js websockets implementation with binary transfer support. I recommend using ws. Afaik. overblown Socket.IO don’t even have binary transfer method.

Server code (node.js):

var WebSocketServer = require('ws').Server;

var wss = new WebSocketServer({
  port: 5555,
  host: '0.0.0.0'
});

// some array for connected sockets

CLIENTS_COUNT = 0;
CLIENTS = { };

wss.on("connection", function(socket) {

  console.log("user connected");

  // store new connection so we can use it later

  CLIENTS[++CLIENTS_COUNT] = socket;
  socket.id = CLIENTS_COUNT;

  socket.on("close", function() {
    console.log("user disconnects");
    delete CLIENTS[this.id];
  });

  socket.on("message", function(message) {
    console.log("message from user", message);
  });
 
  // inform all connected clients about newcomer
  for(var i in CLIENTS) {
    wsSendBinary(CLIENTS[i], {
      message: "new user joins",
      foo: "bar"
    });
  }
});

function wsSendBinary(socket, data) {
  // Encode packet with BiSON to safe up to 50% against JSON.stringify
  var bisonPacket = BISON.encode(data);

  // A tricky part is that you will get nothing sending data as-is.
  // Our bison string is build of chars with indexes from 0 to 255.
  // utf-8 uses extra byte to encode char beyond index 127 so u will end up
  // sending value which can be represented by one byte in two bytes.
  // To take advantage of binary transport we will have to convert our data
  // to javascript typed array. Unsigned Int 8 will do best.

  var uint8Packet = new Uint8Array(packet.length);

  for(var i = 0, len = bisonPacket.length; i < len; i++) {
    uint8Packet[i] = packet.charCodeAt(i);
  }

  // This is how sending binary data looks like with ws library
  socket.send(uint8Packet, {binary: true, mask: true});
}

Client code (browser):

var socket = new WebSocket("ws://yourhost:8080");

socket.onmessage = function(event) {
  // Normally you'd expect that event.data is a string, but in
  // binary transfer u get a write-protected Blob of data
  // which can be read as a stream.

  var reader = new FileReader();

  // There is also readAsBinaryString method if you are not using typed arrays
  reader.readAsArrayBuffer(event.data);

  // As the stream finish to load we can use the results
  reader.onloadend = function() {

    // Another tricky part. Before you can read the results you have to create
    // a view for our typed array
    var view = new Uint8Array(this.result);

    // Now let's decode array containing char indexes to normal ol' utf8 string
    var str = "";
    for(var i = 0; i < view.length; i++) {
      str += String.fromCharCode(view[i]);
    }

    // Our string is still BISON encoded, so last conversion needs to be done.
    var message = BISON.decode(str);                   

    // Voilà, here comes object that we sent
    console.log(message);

    return message;
  };
}