Think about where you have the ladder placed when you push the other block on top of it.
Ok, I was able to solve it, but I am still annoyed about the fact that the ladder blocks don't go up when they can fit in the area that goes up when the button is pressed
