Rewinding a few minutes...
Back then, when Royer talked to the twins about the Astral Tree, he mentioned one thing. The tree was so tall that it pierced through the ceiling of the Golden Palace, going out into the great void.
When the twins' group arrived at this region and saw the Astral Tree for the first time, they understood Royer wasn't kidding. It did indeed break through the ceiling. They could see the enormous cracks that caused some distortions to the 'fake sky' around the tree itself.
Zangeshu had made use of this fact to come up with his idea. "The Golden Palace has existed for so long. Yet, all the Universes waited at the front gate to enter it."
He continued. "That is weird, don't you think? I'm absolutely sure they tried to go around the Golden Palace before, trying to find some other entrance that could bring them further into the Golden Palace. But, from the looks of it, they found nothing. So much so that no one tried it this time or before."
