Xu Feng knew Floral Witch had pulled some scheme, but it was a scheme that he would have done if their position were to be swapped. Hence, he didn't hate Floral Witch for doing such an action.
However, it still left quite a bitter taste in his mouth. No one would willingly become a pawn to be used by others.
Maybe if it had been the "past him" who had no advanced knowledge and was still the stubborn idiot, then he might compromise to become a pawn to ascend. But such a scenario would never happen.
Additionally, his instinct was really shouting at him to disregard the monsters scattered in the field and search for the exit. The instinct was trivial at first, but as time continued to pass, it began to intensify.
This was the main reason why he rejected Floral Witch's invitation.
In fact, he had even stated it somewhat explicitly in which their brain could comprehend. But it was evident that Floral Witch didn't think that way and left to avoid souring the relationship.