Been working on some database import/export infrastructure.
It can get pretty exciting seeing a see each host in a 24 node cluster max out it’s 10Gb/s NIC. Moving data at 28GB/s is pretty exciting.
The speed does come with a drawback, suppose your 24 node bridge cluster talks to 256 database nodes. Now each of those database nodes has a 1Gb/s NIC. Unfortunately one is running at 100Mb/s.
So the crux of parallel processing comes true, you are only as fast as your slowest component.
Back to the export, after the joy of watching the data stream like there is no tomorrow you are left watching one single stream take very close to 35 minutes while everything else finished within 4 minutes.
Try waiting on a bench for 4 minutes then try again for 35 minutes, feels a little different.
Can’t wait for 100GB/s.
1 Trackback or Pingback for this entry:
[...] Filed under: tech — Leave a comment April 27, 2012 As I mentioned in a previous post, I’ve been working on some import/export [...]