Most models work well with Sliding Window attention only, but Gemma uses a mix of global attention & sliding window attention. Cactus forcefully does not allow window size above 1024, which breaks Gemma beyond that context size as it needs global attention to produce coherent text irrespective of context size. Find a work around.