Field notes – ElasticSearch at Petabyte Scale on AWS


I manage a somewhat sizable fleet of ElasticSearch clusters. How large? Well, “large” is relative these days. Strictly in ElasticSearch data nodes, it’s currently operating at the order of:

  • several petabytes of provisioned data-node storage
  • thousands of Xeon E5 v3 cores
  • 10s of terabytes of memory
  • indexing many billions of events a day (24/7/365)

And growing. Individual clusters tend to range anywhere from 48TB to over a petabyte. When I said “petabyte scale”, that includes individual clusters:

Read more at Grey-Boundary