Just a fundamental concept which I struggle to figure out: Why is it necessary that, the software interrupt which triggers context switch, must be of lowest priority? Why should the context switch be postponed until all other interrupt requests are handled?

I think there are other reasons too – but primarily because it is used in interrupt service routines too. The way the interrupt nesting works, to get the most efficient system, and to get best advantage of tail chaining, you want the context switch to only occur once and after all other interrupts have completed – not for a context switch to occur in every nested interrupt which is what would happen if the PendSV interrupt had a priority above that which was requesting the context switch.