The size of EMSK is 8 * tileCount. So each tile in the tileset has 8 bytes, or 64 bits.
The tiles are split up into 64 blocks (8 rows, 8 columns, each covering 4x4 pixels).
Each of the 64 bits flags for one of the 64 blocks of the tile, beginning in the upper left corner, row by row. The bit is set if any of the 16 pixels in the block is transparent. The bit is not set if the whole block is opaque.

EDIT #1: tiles with their INFO byte set to 5 have all bits set.

EDIT #2: (EDI2 haha) About the INFO block: on the ERE it is stated that it contains no byte for the first tile since it's always blank. However, this does not seem to be the case: the length of the block (excluding the fourCC and the two longs in the beginning) is always [numberOfTiles]; it's even always a multiple of 10.

EDIT #3: I've uploaded program output about Castle1, Battle1 and Labrat1. Output begins with "TILE INFO", then all tiles with nonzero INFO bytes (tile# : value) are listed, then "EMSK", followed by a black-and-white visualisation of the EMSK block. Black parts are unset bits, white are set. (NOTE: it's not really an image, but actually a rather large (~4 MB) HTML table. It renders cleanly in Firefox but it's anything but standards complient so it may look buggy in other browsers.)

EDIT #4: JJ2 somehow uses the data in EMSK to determine which parts of the background layers (including layer 4) need to be drawn. Setting all bytes in EMSK to zeroes makes layers 4-8 completely invisible. I couldn't find out how exactly JJ2 uses EMSK to determine this.