Dog organization and motherboard him details job lessons that give offsets of 1 to 2047 are encoded using a 1 byte match. The match length 4 is stored the middle 3 bits of the tag byte. The most significant 3 bits of the offset are stored the upper 3 bits of the tag byte. The lower 8 bits of the offset are stored the next byte. A match overlap the area to be copied. Thus, the string abababa could be written using a literal ab and a match with offset of 2 and length of 5. This would be encoded as: 000001 01100001 000 01 Matches of length 1 to 64 with offsets of 1 through 65535 are encoded using a 2 byte match. The length 1 is encoded the high 6 bits of the tag byte. The offset is stored the next 2 bytes with the least significant byte first. Longer matches are broken into matches of length 64 or less. A 4 byte match allows offsets up to 2 1 to be encoded as with a 2 byte match. The decompresser decode them but the compressor does not produce them because the input is compressed 32K blocks such that a match does not span a block boundary. The entire sequence of matches and literals is preceded by the uncompressed length up to 2 1 written base 128, LSB first, using 1 to 5 digits the low 7 bits. The high bit is 1 to indicate that more digits follow. Compression searches for matches by comparing a hash of the 4 current bytes with previous occurrences of the same hash earlier the 32K block. The hash function interprets the 4 bytes as a 32 bit value, LSB first, multiplies by 0x1e35a7bd, and shifts out the low bits. The hash table size is the smallest power of 2 the range 256 to 16384 that is at least as large as the input string. This is a speed optimization to reduce the time to clear the hash table for small inputs. As optimization for hard to compress data, after 32 failures to find a match, the compressor checks only every second location the input for the next 32 tests, then every third for the next 32 tests, and on. When it finds a match, it goes back to testing every location. As another optimization for the x86 architecture, copies of 16 bytes or less are done using two 64-byte assignments rather than memcpy not find duplicates between versions which a small amount of data was inserted or deleted near the beginning because the data afterward would has moved relative to the block boundaries. This problem can be solved by selecting boundaries that depend on a rolling hash function over a sliding window. We set a boundary whenever n bits of the hash are equal to a fixed value. The average block size be 2 n. For example, the following rolling hash function uses a sliding window of 32 bytes and marks a boundary on average every 16 KB when the high 14 bits of the hash are set to 1. uint32_t h 0; rolling hash value int c; input byte const int K 876543210; a random 32 bit even number not a multiple of 4 while EOF) cumulative counts for ++t+1]; for t+=t; assert; Build linked list int next=calloc); linked list assert; out of memory? for next]++]=i; assert; Traverse and output list for free; bzip2. bzip2 is a popular open source BWT based file compressor developed 1996 by Seward. It takes option -1 through -9 to select a block size of 100 KB to 900 KB. -9 generally gives the best compression. The compression algorithm is as follows: Run length encoding of of zeros. The run length is coded binary LSB to MSB order by two symbols that have values 1 and 2 Runs of length 1 through 10 would be coded as 1, 11, 12, 111, 121. bzip2 uses 2 to 6 Huffman tables, which are selected every 50 symbols to make the code adaptive. The tables are kept a MTF queue. The selection code is unary coded. A unary code for a number n is n 1 bits and a 0. For example, 4. The Huffman tables are coded as a sequence of lengths. The lengths are coded, i.e. as the difference from the previous length. A difference is coded as 0, 10, 11, repeating as needed. A bitmap is used to unused queue selection codes, which