Mechanisms for efficient and responsive distributed applications in compute clusters