Compiling LZ4 in Native ILE on OS400 results in compilation errors on the static asserts regarding the size of LZ4_STREAMDECODESIZE, LZ4_STREAMSIZE and LZ4_STREAMHCSIZE being large enough to cater for 16 byte pointers and the compiler adding implicit padding for field alignment.
Whilst solving the first issue seems relatively straightforward by reorganizing fields in a structure to avoid implicit padding, I am hesitant to suggest how to solve the remaining two issues as the calculation of the values does not seem immediately obvious and comments suggest that there is a potential version to version compatibility impact of changing those values.
1) lz4/lz4-1.9.3/lib/lz4.c line 1426
LZ4_STATIC_ASSERT(LZ4_STREAMDECODESIZE >= sizeof(LZ4_streamDecode_t_internal));
LZ4_STREAMDECODESIZE is set to the size of an array of unsigned long long (8 bytes), on OS400 this array has 6 elements (6 x 8 = 48 bytes)
LZ4_streamDecode_t_internal is defined as follows;
const LZ4_byte* externalDict;
size_t extDictSize;
const LZ4_byte* prefixEnd;
size_t prefixSize;
However to ensure the pointers start on a 16 byte boundary and to ensure the size of the structure is also aligned, implicit padding is added by the compiler after the size_t fields. So the padded size of this structure on OS400 is actually 64 bytes. Provided the order of the fields in the structure is unimportant, reordering the fields like this avoids implicit padding and keeps this structure to 48 bytes as calculated;
const LZ4_byte* externalDict;
const LZ4_byte* prefixEnd;
size_t extDictSize;
size_t prefixSize;
2) lz4/lz4-1.9.3/lib/lz4.c line 2268
A further compile failure on the following;
LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */
On OS/400, LZ4_STREAMSIZE is (1 << 14 + 32) 16416 but sizeof(LZ4_stream_t_internal) is 16488 bytes, again it contains implicit padding to ensure pointer fields are aligned ;
|LZ4_stream_t_internal
|=========================================================
| Offset | Length | Member Name
| Bytes(Bits) | Bytes(Bits) |
|=========================================================
| 0 | 16384 | hashTable[4096]
| 16384 | 4 | currentOffset
| 16388 | 4 | tableType
| 16392 | 8 | ***PADDING***
| 16400 | 16 | dictionary
| 16416 | 16 | dictCtx
| 16432 | 4 | dictSize
| 16436 | 12 | ***PADDING***
==========================================================
But this time even moving the fields around to remove padding, the LZ4_STREAMSIZE would still be short.
3) lz4/lz4-1.9.3/lib/lz4hc.c line 1007
A similar issue for lz4hc.c exists with the following causing a compile error;
LZ4_STATIC_ASSERT(sizeof(LZ4HC_CCtx_internal) <= LZ4_STREAMHCSIZE);
which depends on the following calculations and structure;
#define LZ4_STREAMHCSIZE 262200 /* static size, for inter-version compatibility */
#define LZ4_STREAMHCSIZE_VOIDP (LZ4_STREAMHCSIZE / sizeof(void*))
| Aggregate map for: struct LZ4HC_CCtx_internal
|===========================================================
|...........................................................
|LZ4HC_CCtx_internal
|===========================================================
| Offset | Length | Member Name
| Bytes(Bits) | Bytes(Bits) |
|===========================================================
| 0 | 131072 | hashTable[32768]
| 131072 | 131072 | chainTable[65536]
| 262144 | 16 | end
| 262160 | 16 | base
| 262176 | 16 | dictBase
| 262192 | 4 | dictLimit
| 262196 | 4 | lowLimit
| 262200 | 4 | nextToUpdate
| 262204 | 2 | compressionLevel
| 262206 | 1 | favorDecSpeed
| 262207 | 1 | dirty
| 262208 | 16 | dictCtx
============================================================
The value of LZ4_STREAMHCSIZE would need to be at least 262224 on OS400. The comment suggests that there is an element of inter-version compatibility with the calculation of this number so I'm concerned that there will be a side effect of making a change to the value.
Expected behavior
Clean build of source
To Reproduce
Use CRTCMOD to compile lz4.c and lz4hc.c;
- ftp the lz4-1.9.3.tar.gz to an OS400 machine in binary mode
- Sign in to the OS400 via a 5250 terminal and type
CALL QP2TERM to enter a PASE shell (UNIX-like), navigate to the directory you ftp'd the file to
- Use gunzip and tar utilities to unpack the source code
gunzip lz4-1.9.3.tar.gz;tar -xvf lz4-1.9.3.tar
- Press F3 to leave PASE shell and return to CL command prompt and use CRTCMOD to compile the two modules in error;
CRTCMOD MODULE(LZ4) SRCSTMF('/home/jrumsey/lz4-1.9.3/lib/lz4.c') OUTPUT(*PRINT) OPTION(*FULL)
CRTCMOD MODULE(LZ4HC) SRCSTMF('/home/jrumsey/lz4-1.9.3/lib/lz4hc.c') OUTPUT(*PRINT) OPTION(*FULL)
- Use work with spool files to view the compiler output;
WRKSPLF
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4.c, 1426.5: CZM0232(30) Divisor for modulus or division operator cannot be zero.
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4.c, 1426.5: CZM0104(30) The value of an enumeration constant must be an integral constant expression.
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4.c, 2268.5: CZM0232(30) Divisor for modulus or division operator cannot be zero.
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4.c, 2268.5: CZM0104(30) The value of an enumeration constant must be an integral constant expression.
CZS0601: Module LZ4 is not created because statement errors occurred.
CZM0613: The compilation failed.
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4hc.c, 1007.5: CZM0232(30) Divisor for modulus or division operator cannot be zero.
CZM1003: /home/jrumsey/lz4-1.9.3/lib/lz4hc.c, 1007.5: CZM0104(30) The value of an enumeration constant must be an integral constant expression.
CZS0601: Module LZ4HC is not created because statement errors occurred.
CZM0613: The compilation failed.
System (please complete the following information):
OS/400 V7R3M0 - ILE CRTCMOD compiler
Compiling LZ4 in Native ILE on OS400 results in compilation errors on the static asserts regarding the size of LZ4_STREAMDECODESIZE, LZ4_STREAMSIZE and LZ4_STREAMHCSIZE being large enough to cater for 16 byte pointers and the compiler adding implicit padding for field alignment.
Whilst solving the first issue seems relatively straightforward by reorganizing fields in a structure to avoid implicit padding, I am hesitant to suggest how to solve the remaining two issues as the calculation of the values does not seem immediately obvious and comments suggest that there is a potential version to version compatibility impact of changing those values.
1) lz4/lz4-1.9.3/lib/lz4.c line 1426
LZ4_STATIC_ASSERT(LZ4_STREAMDECODESIZE >= sizeof(LZ4_streamDecode_t_internal));LZ4_STREAMDECODESIZE is set to the size of an array of unsigned long long (8 bytes), on OS400 this array has 6 elements (6 x 8 = 48 bytes)
LZ4_streamDecode_t_internal is defined as follows;
However to ensure the pointers start on a 16 byte boundary and to ensure the size of the structure is also aligned, implicit padding is added by the compiler after the size_t fields. So the padded size of this structure on OS400 is actually 64 bytes. Provided the order of the fields in the structure is unimportant, reordering the fields like this avoids implicit padding and keeps this structure to 48 bytes as calculated;
2) lz4/lz4-1.9.3/lib/lz4.c line 2268
A further compile failure on the following;
LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */On OS/400, LZ4_STREAMSIZE is (1 << 14 + 32) 16416 but sizeof(LZ4_stream_t_internal) is 16488 bytes, again it contains implicit padding to ensure pointer fields are aligned ;
But this time even moving the fields around to remove padding, the LZ4_STREAMSIZE would still be short.
3) lz4/lz4-1.9.3/lib/lz4hc.c line 1007
A similar issue for lz4hc.c exists with the following causing a compile error;
LZ4_STATIC_ASSERT(sizeof(LZ4HC_CCtx_internal) <= LZ4_STREAMHCSIZE);which depends on the following calculations and structure;
The value of LZ4_STREAMHCSIZE would need to be at least 262224 on OS400. The comment suggests that there is an element of inter-version compatibility with the calculation of this number so I'm concerned that there will be a side effect of making a change to the value.
Expected behavior
Clean build of source
To Reproduce
Use CRTCMOD to compile lz4.c and lz4hc.c;
CALL QP2TERMto enter a PASE shell (UNIX-like), navigate to the directory you ftp'd the file togunzip lz4-1.9.3.tar.gz;tar -xvf lz4-1.9.3.tarWRKSPLFSystem (please complete the following information):
OS/400 V7R3M0 - ILE CRTCMOD compiler