Quantcast, an internet audience measurement and ad targeting service, processes over 20 petabytes of data per day using Apache Hadoop and its own custom file system called Quantcast File System (QFS). Today, it’s making that technology available to as open source under an Apache license. You can now find it on GitHub.
The default Hadoop file system is called Hadoop Distributed File System (HDFS). CEO Konrad Feldman says Quantcast started using Hadoop in 2006. In 2008 as Quantcast started collecting 1TB of data per day the team realized that it was going to need a file system with better throughput than HDFS.
They settled on using the open source Kosmos Distributed File System (kosmosfs), but didn’t feel that it as production ready. So they hired Sriram Rao, the lead architect of kosmosfs, to work on making it production ready. The result was QFS, which Quantcast has been using in product for about four years now, though Rao has since left the company for Microsoft.
