zlib is a popular free open-source compression system. Jazz Jackrabbit 2 uses it to compress all the data files, to reduce the space used. This article gives a quick overview of how to use zlib, primarily with the Jazz Jackrabbit 2 files.

zlib basics

Details will be given for C and for Visual Basic 6, though zlib can be used with many other languages. To use zlib with C, you will need zlib.lib and zlib.h, while for Visual Basic you merely need zlib.dll. These files are available from the zlib website, which is given at the end of this article.

Only the compress2 and uncompress functions are required. The crc32 is used for verifying the file, but is not needed.

Declaring zlib functions in C

#include <zlib.h>

Declaring zlib functions in Visual Basic

Don’t forget to add “Private“s in front of them if you’re going to use them in your forms.

Declare Function compress2 Lib "zlib.dll" (ByRef dest As Any, ByRef destLen As Any, ByRef src As Any, ByVal srcLen As Long, ByVal Level As Integer) As Long

Declare Function uncompress Lib "zlib.dll" (ByRef dest As Any, ByRef destLen As Any, ByRef src As Any, ByVal srcLen As Long) As Long

Declare Function crc32 Lib "zlib.dll" (ByVal crc As Long, ByRef buffer As Any, ByVal bufferLen As Long) As Long</pre>

How compression is used in JJ2 files

As a bare minimum, all zlib-compressed files in JJ2 files should at least provide the compressed and uncompressed sizes in its header. For example, consider an example zlib section of a file which goes:

44 00 00 00 4F 00 00 00-78 DA E3 60 80 00 17 73 69 30 ED F0 53 98 41 E0-97 30 03 23 90 DD 92 53 C6 78 E7 87 30 C3 7B 31-47 06 35 89 03 FC 20 F9 04 0A DC DB 3C 07 B0 5A-01 55 39 06 95 04 25 86 2B 40 1A 0C 9E 30 00 00-2B 02 0E A6

In this example, the first four bytes (which read 68 in hex) represents the size of the compressed stream (the input data). The next four bytes (which read 79 in hex) represents the size of the uncompressed stream (the output data). Armed with that data, we now know that we will be decompressing a 68-byte buffer into a 79-byte one. Note that zlib streams starts with a “78 DA” when compressed at the maximum level, and this can be very useful for skimming through large files to find out where a zlib stream begins or how many zlib streams there are.

To decompress a zlib file, the uncompress function is used.

Using uncompress in C

For this short snippet we will assume that you have the above “example” file stored as “tutorial.dat”. This assumes you are writing a console program and have already #included anything necessary.

char *in;
char *out;
int CompressedSize;
int UncompressedSize;

int main() { FILE *fi = fopen(“tutorial.dat”, “rb”); //Opens tutorial.dat for binary reading fread(&CompressedSize, 4, 1, fi); //Reads the first four bytes and stores it fread(&UncompressedSize, 4, 1, fi); //Reads the next four bytes and stores it

in = malloc(CompressedSize); //Allocates a size of CompressedSize for input stream fread(in, 1, CompressedSize, fi); //Reads CompressedSize bytes from the file fclose(fi); //We finished reading the file, no need to use it anymore out = malloc(UncompressedSize); //Allocates a size of UncompressedSize for output uncompress(out, &UncompressedSize, in, CompressedSize); //Decompresses, YAY :) //Here “out” contains the uncompressed data, so do whatever you want with it. //For simplicity, the above example is actually a string, so print it to screen: printf(out); }

Once you can understand the concepts of malloc and uncompress, you can decompress just about every file format in JJ2. Don’t forget to free what you malloc though. Note that we pass a pointer to the second argument of uncompress, which will return with the uncompressed size of the buffer. Assuming you already had the right size in the first place, this value will not change. Also, the return value of uncompress determines whether the operation was a success or not. However, to keep things simple, we assume that our zlib stream is valid, and will not error-handle it.

Using uncompress in VB

Here we use the same assumptions: a file tutorial.dat containing the binary data.

Dim inBuffer() As Byte
Dim outBuffer() As Byte
Dim CompressedSize As Long
Dim UncompressedSize As Long

Sub Main() Open “tutorial.dat” For Binary Access Read As #1 ‘Opens tutorial.dat for reading Get #1, , CompressedSize ‘Reads the first four bytes, stores in CompressedSize Get #1, , UncompressedSize ‘Reads the next four bytes, stores in UncompressedSize

ReDim inBuffer(CompressedSize – 1) ‘Create buffer of CompressedSize bytes Get #1, , inBuffer ‘Reads bytes into inBuffer Close #1 ‘Finish reading from file, no need to use it anymore ReDim outBuffer(UncompressedSize – 1) ‘Create buffer of UncompressedSize bytes Uncompress outBuffer(0), UncompressedSize, inBuffer(0), CompressedSize ‘YAY :) ‘Here “outBuffer” contains the uncompressed data, so do whatever you want with it. End Sub

One thing to note is that ReDim uses “x-1” bytes when you want to allocate x bytes, because the array goes from buffer(0) to buffer(x-1). This is slightly different from how things are in C.

Compressing files

Overall, uncompressing shouldn’t be too difficult, since you already know both the exact compressed AND decompressed size. Trying to compress data however, is more difficult, because you don’t know how big the new zlib stream will be. However, the zlib manual states that the compressed size of a buffer will be at most (1.01 * size + 12) bytes big, regardless of compression level. Here we will use compress2, using a compression level of 9 (maximum).

Using compress2 in C

For this short snippet we will assume that the in buffer already contains a string called “Let us…”. We will compress this into zlib and turn it into a file that is similar to JJ2’s file formats.

char in[] = “Let us attempt to compress this string”;
char *out;
int CompressedSize;
int UncompressedSize;

int main() { UncompressedSize = sizeof(in); CompressedSize = (int)((float)(UncompressedSize * 1.01) + 12); //Allocate space

out = malloc(CompressedSize); //Allocates a size of UncompressedSize for output compress2(out, &CompressedSize, in, UncompressedSize, 9); //Compresses, YAY :) //Note that CompressedSize now holds the real uncompressed size //Here “out” contains the compressed data, so do whatever you want with it. //For our example, we will write this to testcompress.dat FILE *fo = fopen(“testcompress.dat”, “wb”); //Opens testcompress.dat for writing fwrite(out, 4, 1, &CompressedSize); //Writes the compressed size fwrite(out, 4, 1, &UncompressedSize); //Writes the uncompressed size fwrite(out, 1, CompressedSize, fo); //Writes compressed stream to file fclose(out); //We finished writing to the file, no need to use it anymore }

Again, if you are implementing this, remember to #include, and don’t forget to free what you malloc. If you are fussy and want your out buffer to be the right size, use realloc to adjust it.

Using compress2 in VB

Same thing as above, string to zlib, write to testcompress.dat.

Dim inBuffer As String
Dim outBuffer() As Byte
Dim CompressedSize As Long
Dim UncompressedSize As Long

Sub Main() inBuffer = “Let us attempt to compress this string” UncompressedSize = Len(inBuffer) + 1 ‘Add a null-char to the end of the string CompressedSize = UncompressedSize * 1.01 + 12 ‘Sets maximum output size

ReDim outBuffer(CompressedSize – 1) ‘Allocates output space compress2 outBuffer(0), CompressedSize, inBuffer, UncompressedSize, 9 ‘YAY :) ReDim Preserve outBuffer(CompressedSize – 1) ‘Adjusts output space ‘Here “outBuffer” contains the uncompressed data, so do whatever you want with it. ‘For out example, we’ll just do something simple – dump it into testcompress.dat Open “testcompress.dat” For Binary Access Write As #1 ‘Opens file for writing Put #1, , CompressedSize ‘Writes CompressedSize to the first four bytes Put #1, , UncompressedSize ‘Writes UncompressedSize to the next four bytes Put #1, , outBuffer ‘Writes the output buffer to the file Close #1 ‘Finish writing to file, no need to use it anymore End Sub

Again, the Visual Basic function looks slightly harder than C++. Only thing to note is the use of ReDim Preserve. Remember how the UncompressedSize value changes after going through compress2? We adjust the buffer so that it fits exactly the compressed stream, and prevent any problems with Put #1, , outBuffer.

CRC Checking

Finally we get to the crc32 stage. Most of the JJ2 files contain an internal (and some even external) CRC check to verify that the file is not corrupted in any way. Seeing how most programmers are lazy, they will probably skip the CRC mechanism and just assume that file is valid, but it will be included here just for completeness anyway.

The crc32 function takes 3 arguments. The first one is a previous crc value, in case we do the CRC check in chunks. For simplicity, we will always use CRC on the whole buffer at once, which means we leave CRC as 0. buf/buffer is the buffer, and len/bufferLen is the length of the buffer. If you’ve fully understood the uncompress and compress2 functions then there is no need for an example here, since the buffer and length arguments are used here in exactly the same way as before.

Notes

None of the above code has been tested. They were jotted down directly on notepad and have never gone through any stage of compilation.

Related content outside the wiki

Category: