Data - The Need for Speed

• As data grows - finding it and processing it takes longer.
• CPUs became faster, RAM became exponentially larger, we now get exponential growth in disks capacity.
• The bottleneck moved to the disk systems performance.
• Very large disk arrays can handle very high throughput for sequential I/O...
• ...but they are lousy at handling random I/O and inter-dependent I/O:
  • If a disk can handle around 120 random I/O operations per second (IOPS).
  • 100 disks will handle around 12000 IOPS - assuming I/O is split evenly.
  • With fully inter-dependent I/Os (when the result of one will determinate how to send the next) - even 1000 disks won't do more then 120 IOPS.

Originally written by guy keren