General-Purpose Compression via Mathematical Functions (GCMF)

GCMF Binary Format Specification This section defines the binary representation of GCMF. A conforming GCMF decoder MUST be capable of parsing all mandatory structure defined in this specification. Unless otherwise stated, all bit fields are packed from the most-significant bit to the least-significant bit. GCMF is a lossless representation format. A valid GCMF stream MUST decode to exactly the byte sequence represented by the encoded data.

File Structure A GCMF file consists of a fixed header followed by a representation-specific payload. The general structure is: The Function Address determines the interpretation of Representation Data. For mathematical representations, Representation Data consists of the function definition or predefined-function parameters, X_start, X_end, BASE, and, when present, the residual representation. For LITERAL, Representation Data consists entirely of the original source bytes after the required alignment padding. Physical EOF is not encoded as a field or byte. The end of the physical GCMF stream terminates the final representation. The exact fields present after the function address depend on the selected function type. The LITERAL function type is an exception and terminates the structured portion of the stream.

Magic Number Every GCMF file MUST begin with the following 4 bytes: 0x47 0x43 0x4D 0x46. These bytes correspond to the ASCII string "GCMF". A decoder MUST reject a file whose first 4 bytes do not equal 0x47 0x43 0x4D 0x46.

Version Immediately following the magic number is an 8-bit unsigned version field. The version number assigned to this publication is Version 0, written as 0x00. Future versions MAY use additional version values. A decoder encountering an unsupported version MUST NOT attempt to interpret the remainder of the stream according to an unsupported specification.

Function Address The function address is a 5-bit field defined as: where R is the residual flag and AAAA is the function type address.

Residual Flag The most-significant bit MUST be the residual flag, where 0 indicates no residual function is present and 1 indicates a residual function is present. If R = 1, the residual representation immediately follows the primary mathematical representation according to the residual encoding rules in .

Function Type Address LITERAL + RESIDUAL is reserved and MUST be rejected by a conforming decoder unless a future GCMF version explicitly defines semantics for it.

Bit Packing The Function Address occupies exactly five bits and MUST NOT automatically consume a complete byte. The bit immediately following the Function Address is the first bit of the next field unless alignment is explicitly required by the representation. GCMF-VLI units MAY begin at any bit offset within a bit-packed structure. Implementations MUST therefore provide a bit-level reader and writer for structured GCMF fields. Where a field is explicitly required to be byte-aligned, zero-valued padding bits MUST be inserted before that field. Padding bits MUST NOT be interpreted as data and MUST be zero. Unless explicitly specified otherwise, no implicit alignment is performed.

Variable Length Integer Encoding GCMF uses a variable-length integer representation called GCMF-VLI. Each encoded unit contains: where C = 1 indicates that another VLI unit follows, C = 0 indicates the final VLI unit, and VALUE contains 7 bits of the integer. The first unit contains the most significant portion of the value. Therefore, GCMF-VLI is a big-endian base-128 representation. A decoder MUST continue reading VLI units until a unit with C=0 is encountered. A VLI MUST NOT contain unnecessary leading zero payload groups.

Signed Integer Encoding Where a mathematical parameter may be negative, GCMF uses ZigZag encoding before GCMF-VLI encoding. For a signed integer n: = 0 Z(n) = -2n - 1, n < 0 ]]> The resulting non-negative integer is then encoded using GCMF-VLI. This allows small negative and positive values to have similarly compact representations.

CUSTOM Function A CUSTOM function has Function Type Address: or, when residual data is present: The structure is: The Function Length specifies the number of bits occupied by the function bytecode, including opcode bits, operand bits, and the END instruction. The Function Length MUST be sufficient to contain a complete bytecode sequence ending in an END instruction. The decoder MUST NOT read beyond the declared function length while executing the function. Any unused bits between the final END instruction and the declared function boundary MUST be zero. A canonical encoder SHOULD emit no unused bits after END. The function bytecode MUST contain a valid END instruction within the specified function length.

Predefined Functions A predefined function does not contain its mathematical definition in the file. The Function Address identifies the mathematical function through the GCMF predefined-function registry. The decoder obtains the function definition from the version-specific registry. A predefined function therefore has the general structure: Each predefined function MUST have a precisely defined parameter encoding.

Predefined Function Parameter Encoding Unless otherwise specified, all integer parameters of predefined functions are encoded using GCMF-VLI. Parameters that may be negative MUST use ZigZag encoding before GCMF-VLI encoding. The predefined functions use the following parameter encodings: For MODULAR, G(x) is a mathematical function defined by a nested GCMF function representation and m is a positive integer modulus. The nested function G(x) MUST be encoded using the same function representation rules defined by this specification, excluding residual encoding. The modulus m MUST be encoded as an unsigned GCMF-VLI integer and MUST satisfy m >= 1. For every domain value x, the decoder first evaluates G(x) and then computes: The result of the modulo operation MUST be the unique integer r satisfying 0 <= r < m, where G(x) = qm + r for some integer q. The MODULAR function MUST NOT contain a residual representation within its nested G(x) function. Recursive MODULAR functions MAY be permitted, provided that the implementation-defined maximum nesting depth is not exceeded. A decoder MUST reject a MODULAR representation if G(x) is invalid, m is zero, or evaluation of G(x) produces a value outside the supported numerical domain. All parameters MUST appear in the order specified above. A decoder MUST reject a representation containing missing, malformed, or unexpected parameter data.

Function Bytecode GCMF mathematical functions use a stack-based bytecode. Each opcode occupies exactly five bits. The opcode is immediately followed by its operand, if the opcode requires one. No implicit byte alignment is performed between an opcode and its operand. For CONST, the operand is a signed GCMF-VLI value. For instructions that do not require an operand, no operand bits are present. The Function Length counts every bit occupied by the bytecode, including five-bit opcodes and all operand bits. A decoder MUST NOT read beyond the declared Function Length while decoding instructions. The bytecode MUST contain exactly one terminating END instruction as its logical terminator. A canonical bytecode representation MUST NOT contain instructions after END. Because opcodes are bit-packed, an operand begins at the bit immediately following the opcode.

Stack Machine Requirements A custom function is evaluated independently for every value of x. Before evaluating a new x, the evaluation stack MUST be empty. Each instruction operates on the stack. For binary operators: [a, b] becomes [a OP b]. For unary operators: [a] becomes [OP(a)]. DUP transforms: [a] becomes [a, a]. X pushes the current domain value. CONST pushes its encoded constant. END terminates execution. At END, the stack MUST contain exactly one value. That value is the output of the function. A function that underflows the stack; executes an invalid operation; reaches the declared function boundary without END; contains an invalid operand; or terminates with anything other than exactly one stack value MUST be considered invalid.

Numerical Evaluation GCMF v1 mathematical evaluation MUST use exact integer arithmetic unless a specific operation is explicitly defined to use another numerical domain. An operation whose mathematical result is undefined within the applicable numerical domain MUST cause decoding to fail. Examples include division by zero, logarithms outside their valid domain, square roots of invalid values, and other undefined operations. A decoder MUST reject a representation if evaluation produces a value that cannot be represented by the numerical domain defined for the representation. Implementations MUST NOT silently round, truncate, saturate, wrap, or otherwise alter a mathematical result unless such behavior is explicitly defined by the applicable GCMF version. All conforming implementations of the same GCMF version MUST produce the same mathematical result for the same valid input.

Example Custom Function The function f(x) = 3x + 10 can be represented as: The logical execution is: [3] X -> [3, x] MUL -> [3x] CONST 10 -> [3x, 10] ADD -> [3x + 10] END -> result ]]>

X_start and X_end For all non-LITERAL representations, the domain is specified using two unsigned GCMF-VLI fields: X_start and X_end MUST be non-negative integers. The default domain is the integer sequence: The number of generated values is: X_end MUST be greater than or equal to X_start. A decoder MUST reject a representation whose domain would require an invalid, negative, or implementation-prohibited number of values. The number of generated values MUST also be consistent with the number of source values represented by the GCMF stream.

Base Encoding The BASE field specifies the numerical radix associated with the domain representation. BASE is encoded as an unsigned GCMF-VLI integer. The base MUST be greater than or equal to 2. A conforming implementation SHOULD support at least bases 2 through 36. The BASE field does not alter the binary encoding of GCMF-VLI values, function opcodes, function bytecode, or other binary fields.

Base Interpretation BASE specifies the radix used for the interpretation of X_start and X_end when a representation requires base-dependent numerical interpretation. The mathematical values represented by X_start and X_end are independent of the selected radix. For example, a domain value of decimal 16 has the same mathematical value regardless of whether BASE is 10 or 16. BASE therefore MUST NOT change the numerical value of X_start or X_end. An encoder MAY evaluate multiple BASE values when selecting a representation if the selected base affects the representation length of the domain under the encoding rules defined by this specification. If BASE does not affect the encoded length or interpretation of a candidate, changing BASE MUST NOT be used as a basis for selecting that candidate.

Residual Representation If the Residual Flag is 1, a residual representation follows the primary function representation. The residual represents the difference between the original datum and the primary mathematical output. For every value: The original datum is reconstructed using: The residual representation MUST itself be encoded using a valid GCMF function representation. Therefore a residual MAY be a predefined function, a custom function, or another mathematically represented sequence, subject to the restrictions imposed by the GCMF version. The residual representation does not independently require a second Function Address byte unless specified by the residual container structure.

Residual Function Container When R = 1, the residual representation immediately follows the complete primary mathematical representation. The residual section is encoded as: The Residual Function Type uses the same function-type address table as the primary function, excluding the residual flag. The residual function therefore contains exactly four function-type bits. For a custom residual, 0000 is used as the Residual Function Type. A custom residual MUST contain its Function Length followed by its Function Bytecode according to the CUSTOM function encoding rules. A predefined residual MUST contain the parameter fields specified for its function type. The residual function MUST NOT contain a residual flag and MUST NOT contain another residual representation. Recursive residual nesting is therefore prohibited in GCMF v1. The residual representation MUST generate exactly the same number of values as the primary representation.

Literal The LITERAL function has Function Type Address: and MUST NOT have the Residual Flag set. Its complete Function Address is therefore: LITERAL is a terminal representation. Immediately following the Function Address, the decoder MUST consume three zero-valued padding bits. These three padding bits align the stream to the next byte boundary. The remainder of the GCMF stream is then interpreted as raw source bytes. No function length is stored. No X_start is stored. No X_end is stored. No BASE field is stored. No residual representation is stored. The decoder MUST copy the literal payload directly to the output without interpretation or transformation.

Physical End Of File The physical end of the GCMF file is the termination condition for the final field. No EOF byte or other in-band EOF marker is defined by GCMF. This is particularly important for LITERAL, because every byte value from 0x00 through 0xFF is valid source data. Consequently, no byte value MAY be reserved as an EOF marker. For LITERAL, EOF is the physical end of the GCMF stream. The final byte of the file is therefore the final literal source byte. A transport or container system embedding GCMF MUST preserve the exact file length.

Complete Canonical Layout For a normal mathematical representation without residual data: For a representation containing a residual: For LITERAL:

Encoding Example Consider the sequence: It can be represented by the linear function f(x) = 3x + 10 over the domain X = 0, 1, 2, 3, 4, 5. The logical GCMF representation is therefore: The decoder evaluates: The resulting sequence exactly reproduces the source sequence.

Encoder Selection Algorithm The encoder SHOULD construct multiple candidate representations. Candidates MAY differ in function type, function parameters, custom bytecode, domain, numerical base, or residual representation. For every candidate C, the encoder computes L(C), the total encoded length in bytes. The encoder selects: The encoder MUST include all structural overhead in this calculation. Therefore the cost of a candidate includes magic, version, function address, function length where applicable, function bytecode, parameters, domain, base, residual data, and any required alignment. The encoder MUST NOT select a mathematical representation merely because the function itself is shorter.

Literal Selection The encoder MUST be permitted to select LITERAL when no mathematical representation provides sufficient compression. The encoder SHOULD compare the best mathematical candidate against the LITERAL representation: This prevents the creation of deliberately inflated mathematical representations for incompressible data.

Decoder Validation A decoder MUST reject a GCMF stream if any of the following occurs:

The magic number is invalid.
The version is unsupported.
The Function Address is invalid.
A required VLI is truncated.
A VLI is malformed.
A custom function exceeds its declared length.
A custom function lacks a valid END.
A function causes stack underflow.
A function terminates with an invalid stack state.
X_end is less than X_start.
BASE is less than 2.
A required function parameter is missing.
A residual representation is malformed.
A prohibited recursive residual is encountered.
The mathematical evaluation produces a value that cannot be represented by the selected numerical representation.
The stream terminates before all mandatory fields are read.

A decoder SHOULD also impose implementation-defined limits on:

maximum function length;
maximum domain size;
maximum stack depth;
maximum numerical magnitude;
maximum execution time.

Such limits are important because GCMF functions are executable mathematical expressions and should not be permitted to consume unbounded computational resources.

Canonical Encoding A GCMF encoder SHOULD produce canonical encodings. A canonical representation MUST:

use the shortest valid GCMF-VLI representation;
avoid unnecessary leading VLI groups;
use the minimum required function length;
terminate bytecode with exactly one END;
avoid unnecessary residual representations;
avoid unnecessary domain extension;
avoid unnecessary padding;
select the smallest valid complete representation when deterministic encoding is required.

Canonicalization permits byte-for-byte comparison of equivalent GCMF representations.

Versioning The version byte identifies the binary interpretation rules. A future version MAY:

add predefined functions;
add bytecode instructions;
modify residual semantics;
add new representation modes;
define previously reserved fields.

Existing function addresses MUST NOT be silently reassigned within a compatible version. A new incompatible interpretation MUST use a new version number.

Reserved Space The initial GCMF function address space has no unused base-function address because all 16 four-bit values are assigned. Future extensions SHOULD therefore use one of the following mechanisms:

a new GCMF version;
extension instructions;
additional extension containers;
a future extension mechanism explicitly defined by the specification.

The five-bit address space itself MUST NOT be silently expanded within GCMF v1. The bytecode instruction space is similarly fully allocated.

Losslessness Requirement For every valid GCMF representation G, decoding MUST satisfy: where D is the exact original input byte sequence. For mathematical representations: For exact mathematical representations, r(x(i)) = 0. For LITERAL: No rounding, approximation, or numerical loss is permitted unless explicitly introduced by a future GCMF version.

Summary of Binary Grammar The logical grammar of GCMF v1 is: The fundamental mathematical decoding equation is: where r(x(i)) = 0 for representations without residuals. For LITERAL, Decode(G) = LiteralPayload(G). No additional EOF field is encoded.