Facebook open sources Zstandard compression algorithm and MyRocks storage engine
Today, Facebook is releasing its Zstandard compression algorithm into the wild as open source. The lossless compression technology is aimed at replacing existing libraries like zlib that are powered by the outdatedDeflate compression algorithm. In addition to Zstandard, Facebook is also dropping its MyRocks storage engineas open source. MyRocks is currently being used by Facebook to improve the storage efficiency of its MySQL databases.
Both releases occurred in coordination with Facebooks @Scale conference in San Jose. The conferencesupports engineers working to solve large-scale engineering challenges by bringing them together, side-by-side, with other companies in the industry dedicated to increasing the prevalence of open-source technology. Both Zstandard and MyRockswill enable developers to build platforms that can efficientlyscale to large and diverse user bases.
One of the greatest fears of engineers looking to update and replace critical infrastructure is that the new libraries could bring an entire service down. Jay Parikh, vice president of infrastructure engineering at Facebook, is proud of the fact that both Zstandard and MyRocks weretested at scale acrossthe company before being released to the community.
Everyone here uses all of the products, added Parikh.
In the six months that Facebook has been testing Zstandard, the company noted impressive results.With the same compression ratio of zlib, Zstandard was five times faster. In a case where thecompression time was held constant, files were 10 percent smaller.
MyRocks has also produced some serious storage efficiencies. When compared to compressed InnoDB, MyRocks was able to store the same amount of data on half the server space.
Making it open source makes the community better, added Parikh. It will be adopted more quickly as open sourcethan had if wascreated as the proprietary technology of a single company.
Facebook hopes that their open source solutions can become new industry standards.