On the early 90's I remember a lecture at college
where about a distributed system of Unix servers taking over the responsibility of a Cray supercomputer - apparently the only reason the Cray wasn't disconnected was for the tape library it controlled.
So then I believed Cray were a dying breed of dinosaurs. I was wrong, they are alive and kicking, and you can get one for the price of a regular server 10 to 15 years ago.
I never had so much need for computational power has I did on project SADPOF, an operational decision system for the forest. I'm talking about Simulated Annealing with multi-objective evaluation, huge parallel processing needs of huge chucks of data, with inner LP solving problems. When I'm talking about huge data, I'm talking about simulation variables of matrixes of floats of 6000 x 36 x 32 x 15. And calculating about 30 complex restrictions and a huge objective function.
Our test box was a 2x Quad core and it costed about $5K. A Cray would be welcomed, though MPI doesn't seem to much appealing to work with. What seems appealing is PLINQ, which will no doubt updated to work on HPCs using MPI.