
If we’re going to talk about truly BIG data, we need to look to China. With 1.3 billion people, a quickly expanding urban economy, and rising rates of internet and smartphone penetration, China generates a mind-boggling amount of data annually.
As just one example, I spoke to Baidu, China’s search giant, about how it crunches petabytes of data every day for analytic queries in under 30 seconds. It can move data at memory speeds using new virtual distributed storage software called Alluxio, formerly Tachyon, that was originally developed at UC Berkeley’s AMPLab.
I recently caught up with Lei Xu, big data platform senior engineer of Qunar, China’s top e-commerce travel site that carries more than 80% of online travel business in the Middle Kingdom. I was curious about how their big data backend keeps up with their business’ exploding performance and scale requirements.
TechRepublic: First, please describe the scale of your big data operations at Qunar.
Xu: Our streaming platform processes around six billion system log entries daily; that’s more than 4.5 terabytes of data. Every day. And it’s growing quickly. One example of a critical job running on this platform is our real-time user recommendation engine. Stability and low latency are everything. It’s based on the log analysis of users’ clicks and search behavior. Faster analysis gives our users more accurate feedback.
TechRepublic: How did you architect your original backend for processing big data and what were the problems you ran into as more users hit your site?
Xu: Our real-time stream processing systems uses Apache Mesos for cluster management. We use Mesos to manage Apache Spark, Flink, Logstash, and Kibana. Logs come from multiple sources and we consolidate them with Kafka. The main computing frameworks, Spark streaming and Flink, subscribe to the data in Kafka, process the data, and then persist the results to HDFS [Hadoop Distributed File System].
But this architecture had some serious bottlenecks.
First, the HDFS storage system was located in a remote cluster, adding major network latency. Data exchange became a choke point. Because HDFS uses spinning disks, I/O operations (especially write) have high latency, too. Spark streaming executors need to read from HDFS and the repetitive cross-cluster read operations really deprecate performance.
Next, if a Spark streaming executor runs out of memory and fails, it will restart on another node but won’t be able to reuse the checkpoint information from the original node. That means some jobs can never finish.
Finally, when streaming tasks persist data using MEMORY_ONLY, you replicate data in JVM memory causing garbage collection issues and even failure. Let’s say you get clever and just use MEMORY_TO_DISK or DISK_ONLY, then the whole performance processing architecture is capped by the speed of disk I/O. We needed more speed.
TechRepublic: How did you remove these bottlenecks in your new architecture?
Xu: The original stream processing logic remains unchanged in the new architecture, but we deployed Alluxio between the computation frameworks and HDFS. In this architecture, only the final results need to be persisted to the remote HDFS cluster. The computing frameworks exchange data via Alluxio in real-time processing at memory speed data access.
The main difference before and after is that much of the data used and generated by Spark streaming is stored in Alluxio, eliminating communication with the slow, remote HDFS cluster. Flink and Zeppelin and other computing frameworks also access the data stored in Alluxio at memory speeds.
There were several key features in Alluxio that helped us dramatically improve overall system performance and reduce latency:
First, Alluxio uses a tiered storage approach. Alluxio runs on every compute node to manage the local memory, SSD, and disk. Data required by the stream processing is saved locally as much as possible, reducing network bandwidth. Alluxio applies a clever eviction notice to store data in memory that is used most often.
Unified namespace greatly simplifies input and output logic. Alluxio unifies the HDFS management and provides a global, unified namespace for all computation frameworks. The other important feature is a simple API with easy integration. It’s truly easy to convert applications running on HDFS to Alluxio by swapping in “Alluxio” for “hdfs” in the URIs. That’s it!
Finally, Alluxio off-heap storage helps with Java garbage collection. Spark streaming jobs store data in Alluxio instead of Spark’s executor’s JVM. This keeps the benefits of in-memory performance, while greatly reducing Java garbage collection costs. It also prevents out-of-memory errors caused by redundant in-memory data blocks in Spark executors.
TechRepublic: What kind of payoff did you see by changing architecture?
Xu: Performance was the huge payoff. Our new system is much faster. We saw a 15X speedup on average with a 300X speedup at peak service times. Throughput increased from about 20 to 300 events per second to a stable throughput of 7,800 events per second. Users on our travel site love the faster response times.
This article was originally published on www.techrepublic.com can be viewed in full


Archive
- October 2024(44)
- September 2024(94)
- August 2024(100)
- July 2024(99)
- June 2024(126)
- May 2024(155)
- April 2024(123)
- March 2024(112)
- February 2024(109)
- January 2024(95)
- December 2023(56)
- November 2023(86)
- October 2023(97)
- September 2023(89)
- August 2023(101)
- July 2023(104)
- June 2023(113)
- May 2023(103)
- April 2023(93)
- March 2023(129)
- February 2023(77)
- January 2023(91)
- December 2022(90)
- November 2022(125)
- October 2022(117)
- September 2022(137)
- August 2022(119)
- July 2022(99)
- June 2022(128)
- May 2022(112)
- April 2022(108)
- March 2022(121)
- February 2022(93)
- January 2022(110)
- December 2021(92)
- November 2021(107)
- October 2021(101)
- September 2021(81)
- August 2021(74)
- July 2021(78)
- June 2021(92)
- May 2021(67)
- April 2021(79)
- March 2021(79)
- February 2021(58)
- January 2021(55)
- December 2020(56)
- November 2020(59)
- October 2020(78)
- September 2020(72)
- August 2020(64)
- July 2020(71)
- June 2020(74)
- May 2020(50)
- April 2020(71)
- March 2020(71)
- February 2020(58)
- January 2020(62)
- December 2019(57)
- November 2019(64)
- October 2019(25)
- September 2019(24)
- August 2019(14)
- July 2019(23)
- June 2019(54)
- May 2019(82)
- April 2019(76)
- March 2019(71)
- February 2019(67)
- January 2019(75)
- December 2018(44)
- November 2018(47)
- October 2018(74)
- September 2018(54)
- August 2018(61)
- July 2018(72)
- June 2018(62)
- May 2018(62)
- April 2018(73)
- March 2018(76)
- February 2018(8)
- January 2018(7)
- December 2017(6)
- November 2017(8)
- October 2017(3)
- September 2017(4)
- August 2017(4)
- July 2017(2)
- June 2017(5)
- May 2017(6)
- April 2017(11)
- March 2017(8)
- February 2017(16)
- January 2017(10)
- December 2016(12)
- November 2016(20)
- October 2016(7)
- September 2016(102)
- August 2016(168)
- July 2016(141)
- June 2016(149)
- May 2016(117)
- April 2016(59)
- March 2016(85)
- February 2016(153)
- December 2015(150)