Original Architecture

Initially, Tencent’s game logs were stored in a Hadoop File System, while application-layer data was dispersed across traditional databases like MySQL and PostgreSQL. Real-time data was stored in Druid. This storage scheme made data usage difficult and often created bottlenecks in accessing data.

Managing two separate data pipelines for real-time and batch was complex and costly. For querying, all data first had to be pre-processed, including pre-aggregation and denormalization in Hive, before moving to PostgreSQL for reporting and dashboards. This process was complex and a waste of computing and storage resources, locking the data into a rigid single-view format: any schema change required reconfiguring the pipeline and data backfilling.

Challenges

Faced with the growth of trillions of data points daily, the gap between the scale of data and cluster capacity widened. A new system was greatly needed. The new system required:

• Sub-second query latency

• The ability to store and manage petabytes of data

• Low operational overhead: With simple operation and maintenance

The New Architecture

After researching for different solutions, Tencent found that Iceberg could meet their needs. Iceberg is an open table format for analytics that supports schema evolution, hidden partitioning, time travel, and version rollback. These features enable data warehouse workloads to be unified on a single copy of data in open formats, allowing for easy data governance and simple architectures.

To achieve second-level query and data freshness, Tencent uses StarRocks as the query engine and the real-time ingestion layer. StarRocks is a blazing-fast, massively parallel processing query engine with features such as:

• Fully vectorized engines

• High concurrency support

• Real-time analytics on mutable data

• A simple architecture: consisting of only two types of processes: FE (Front End) and CN (Compute Node)

With Apache Iceberg and StarRocks, Tencent Games was able to develop a lakehouse architecture with the following capabilities:

• Simple and scalable architecture for trillions of records of new data per day

• Second-level query latency on petabytes of data

• Second-level data freshness with mutable data persisted in Apache Iceberg

Here are some additional features and optimizations that Tencent Games has built to achieve these impressive results.

Cloud Native

Tencent Games developed a stateless CN (compute node) and a Kubernetes operator. Being stateless, CNs can be dynamically scaled to accommodate the change in workload, which saves significant computing resources for Tencent Games’ dynamic workloads.

Second-level data freshness with mutable data on Apache Iceberg

All data is stored in Iceberg, and the CN is used to achieve elasticity and resource isolation. Hot data is temporarily stored in the StarRocks BE nodes and periodically sunk to Apache Iceberg for data persistence. Powered by StarRocks’ storage engine (BE nodes), Tencent Games can easily get second-level data freshness with mutable data. A single source of truth data is also preserved with all data persisted in Apache Iceberg.

For querying, StarRocks tables are utilized as the single point of access for both data in StarRocks and Iceberg. User’s queries are automatically rewritten by an RBO to query data from both StarRocks and Apache Iceberg seamlessly.

Addressing Challenges

After running the aforementioned lakehouse architecture in production, Tencent Games encountered several challenges:

• Waste of resources: Large query requests during peak periods require significant computing resources, leading to waste during low periods.

• Poor resource isolation: SQL queries for large data in Iceberg consumed significant CPU and memory resources, causing other requests or data imports to fail or slow down.

• Slow execution plan generation: Continuous real-time ingestion generates millions of data files, making the query plan generation process slow. It took more than 65 seconds to generate a plan for a query that scans 2 months of data.

Functional Optimizations

To address these challenges, Tencent Games implemented the following optimizations:

Elastic Scaling Tencent Games developed a Kubernetes operator for stateless compute nodes (CN). With compute nodes running as containers in Kubernetes, it enables easy deployment, maintenance, and dynamic scaling to reduce resource waste.

Physical Resource Isolation Using Kubernetes, Tencent Games can spawn groups of Compute Nodes (CNs) on demand with identical labels but different specifications — for example, group 1 with 4 cores and 8GB RAM, and group 2 with 24 cores and 64GB RAM. This setup allows for customization in the number of CNs per group and offers independent lifecycle management, where groups can be destroyed post-task. Importantly, this architecture enables physical resource isolation for Tencent Games’ heterogeneous workload.

Caching Immutable Meta Files Tencent Games also added an LRU FileIO cache within a customized Hive Catalog to more efficiently access immutable metafiles including metadata files, manifest lists, and individual manifest files. This optimization significantly reduces I/O overhead, speeding up metadata access.

Optimizing Iceberg’s Execution Plan When there are too many data files, reading column statistics for all of them is a very resource-intensive task, especially during query planning. Tencent Games pre-calculates and caches column statistics of data files in StarRocks tables, which can be read in parallel by CN and BE nodes to further accelerate the execution plan generation process.

Results

These optimizations brought significant benefits — a query plan that used to take 65 seconds to generate now only takes 6; the stateless CNs also saved tens of thousands of computing cores. Additional Apache Iceberg optimizations such as snapshot cleanup and data file compaction further improved the performance and manageability of the lakehouse architecture.

Q&A

Q: What is the technology for sub-second response times for complex queries? A: Hongli: Use StarRocks as a warehouse and as a cache for the lake caching hot data. For lake queries use elastic computing resources.

Q: What workloads are you putting onto StarRocks versus Spark versus Trino, and why? Any performance comparison to share between StarRocks, Spark, and Trino? A: Hongli: Because a warehouse is needed for cache, there is no comparison with Spark. Comparing StarRocks and Trino on data lake in our scenario, there is a 10x improvement. That’s quite impressive. Adding the new flexible compute nodes (CN) makes StarRocks more flexible than Trino.

Q: Are you using Velox open source for C++ SIMD vectorized engine? A: Hongli: We did it ourselves. (StarRocks SIMD vectorized engine is implemented from scratch)

Q: Are statistics per partition? A: Hongli: Each data file.

Q: Pre-calculated cached data are put in your own set of metadata? A: Hongli: We put the statistical information of data files into a table (StarRocks tables) in a warehouse.

Q: Did you have trouble keeping the column statistics materialization up to date for optimizing query planning? A: Hongli: It went relatively smoothly. Real-time data still needs to be calculated.

Q: What is the maintenance cost comparing Trino and StarRocks? A: Hongli: StarRocks has only two components. The elastic nodes are deployed on Kubernetes, and maintenance is also very easy.

Q: Are most of the features mentioned here available to open-source StarRocks or just internally available in Tencent? A: Hongli: The functions we developed have also been contributed to the community, such as elastic node CN. The feature of data syncing into the (Iceberg) lake is still being contributed.

Join Us on Slack

If you’re interested in the StarRocks project, have questions, or simply seek to discover solutions or best practices, join our StarRocks community on Slack. It’s a great place to connect with project experts and peers from your industry. You can also visit the StarRocks forum for more information.

WeChat is the world’s largest standalone mobile app, serving over 1.3 billion monthly active users as a platform for instant messaging, social media, and mobile payments. To support its unprecedented and rapidly expanding user base, WeChat’s technological backend has had to evolve quickly as well, transitioning from a Hadoop + data warehouse architecture to a modern open data lakehouse architecture.

Challenges

With over a billion users, it comes as no surprise that WeChat manages extremely large data volumes. In some cases, single tables are growing by trillions of records daily and queries regularly scan over 1 billion records.

WeChat’s business scenarios demand rapid end-to-end response times, with a query latency P90 target of under 5 seconds, and data freshness requirements that vary from seconds to minutes. This complexity is elevated by the need to process often more than 50 dimensions and 100 metrics at a time.

WeChat’s legacy data architecture involved a Hadoop-based data lake system along with a variety of data warehouses. This resulted in significant operational overhead and data governance challenges including:

• Juggling multiple systems from separated real-time and batch analytics pipelines

• Maintaining data ingestion pipelines for data warehouses

• Governance challenges from managing multiple copies of the same data

• Managing incompatible APIs of different systems

• Challenges in standardizing data analysis processes

To address these issues, WeChat pursued a unified approach to their analytics which necessitated a redesign of their data architecture.

Solution

WeChat’s revamped Data Lakehouse architecture now features StarRocks as the low latency query engine with Apache Spark as its batch processing engine. Data is stored as Parquet files on Tencent Cloud’s COS (cloud object storage) with Apache Iceberg as the data lake table format.

WeChat’s new StarRocks-based architecture

This new architecture supports both real-time and near-real-time data ingestion:

• For real-time ingestion: data is first ingested into a warehouse in real-time, then cold data is sunk into the data lake and queries can automatically union cold and hot data.

• For near-real-time ingestion: raw data is directly ingested into Apache Iceberg, and then cleaned and transformed using StarRocks’ materialized view.

Result

WeChat’s StarRocks-based data lakehouse solution is now in production across multiple business scenarios within the company including livestreaming, WeChat Keyboard, WeChat Reading, and Public Accounts.

By unifying all workloads with one system, WeChat has experienced significant operational benefits from this improved efficiency. Their live streaming business is one example: the new lakehouse architecture halved the number of tasks data engineers are required to manage, reduced storage costs by over 65%, and shortened the development cycle of offline tasks by two hours.

Not only is their architecture now simplified, data freshness and query latency improved as well, with batch ingestion being eliminated and their near-real-time data pipeline and query latencies being brought down to the mostly sub-second level.

Table 1: WeChat’s data lakehouse performance numbers

What’s Next For WeChat

Looking ahead, WeChat’s goal is to continually explore and refine their existing data lakehouse architecture to further integrate it across more critical operations including:

• Enabling users to interact with the system using SQL without needing to understand the underlying architecture.

• Unifying data access, querying, and the storage system.

• Unifying SQL interaction standards across the platform.

Download a PDF of This Use Case

Join Us on Slack

If you’re interested in the StarRocks project, have questions, or simply seek to discover solutions or best practices, join our StarRocks community on Slack. It’s a great place to connect with project experts and peers from your industry. You can also visit the StarRocks forum for more information.

Originally published at https://www.starrocks.io/blog/how-wechats-data-lakehouse-architecture-enhances-efficiency-for-trillions-of-daily-records

In this article, we'll examine how to overcome slow query performance, a common yet critical issue for data lakehouses that's the source of these broken promises. We'll then explore five data lakehouse architectures from industry leaders that showcase how enhancing your query performance can lead to more than just compute savings - it will simplify data architecture, strengthen governance, and accelerate development cycles too.

**

How Bad Query Performance Breaks the Promise of the Lakehouse**

Despite functional similarities with data warehouses, data lakehouses often lag in performance, which is caused by data lake query engines not being optimized for high concurrency, low latency workloads. This has forced users to copy their data from the lakehouse into proprietary data warehouses to achieve their desired query performance—through a complex, costly ingestion pipeline that undermines data governance and data freshness.

• Cost #1: Data ingestion is expensive. Writing data into the proprietary format required by your data warehouse consumes substantial hardware resources and storage redundancy. Beyond just hardware, the process is labor-intensive. Seemingly simple tasks, like matching data types across systems and table/index design in the data warehouse, are resource-heavy. Plus, this data ingestion inevitably adds latency, compromising data freshness.

• Cost #2: Data ingestion compromises data governance, the very thing that the data lakehouse promises. When there are multiple copies of the same data, ensuring consistent updates across all copies, avoiding discrepancies, and maintaining overall strong data governance are substantial challenges. These aren't mere theoretical concerns but practical issues needing considerable engineering work. If not handled properly, they can jeopardize the reliability of your data-driven decisions.

What Should You Do Instead?

Many enterprises are seeking alternatives that address the shortcomings of their traditional data lake query engines and the added expense of their proprietary data warehouses. Fortunately for these enterprises, modern query engines, optimized for low-latency workloads on open data lakes, now make it feasible to run demanding tasks directly on the data lakehouse.

Let's take a look at the key technologies that are making data warehouse performance on open data lakes possible:

• MPP with in-memory data shuffling: This approach uses massively parallel processing engines with in-memory data shuffling, bypassing disk persistence used by batch analytical engines to reduce latency and improve real-time query performance.

Map Reduce vs. MPP structure

• Integrated caching frameworks: A hierarchical caching system, combining disk-based and in-memory caching, is integrated within the query engine. This effectively reduces and stabilizes data retrieval latency by shielding it from the unstable and low performance of data lake systems.

These improvements, along with system-level optimizations like SIMD, ensure even the most demanding queries can run on an open data lake with ease.

Beyond Fast Raw Query Performance: On-Demand Pre-computation Pipelines

Accelerating data lake queries often involves pre-computation pipelines, where processing tools like Spark are used to create denormalized and pre-aggregated tables. This approach, aimed at minimizing runtime JOINs and aggregations, poses challenges for both data engineers and users.

Data users must adjust their SQL queries to access these pre-computed tables, which is a tedious task. This forces platform engineers to orchestrate pre-computation pipeline demand before downstream applications are developed. This often leads to the creation of underutilized pre-computed tables, resulting in wasted resources, extended development cycles, and increased costs.

Query optimization process

As query engines have evolved, there's been a shift in data processing strategies. Originally, due to slower query engines, there was a heavy reliance on pre-computing everything. Now, with faster query engines, the focus has shifted to on-demand pre-computation. This new requirement for flexibility and speed creates a mismatch with traditional pre-computation pipelines, which weren't designed for such on-the-fly computational needs.

New solutions have been designed for modern query engines to embrace on-demand pre-computations, one of which is an advanced materialized view with query rewrite capabilities. Read more about it here: How to Seamlessly Accelerate Data Lake Queries

5 Architectures to Transform Your Lakehouse

This all might sound good in theory, but how does it work against actual production workloads? Let's use five different data lakehouse architectures being employed by industry leaders to see how this all comes together:

• A leading social media company has shortened its development cycle and improved cost-effectiveness for its trillions of daily records of data by switching to a data lakehouse architecture. Read the case study.

• A gaming giant is reducing storage costs by 15x while eliminating all pre-aggregations through unifying all workloads on its data lakehouse. Read the case study.

• A leading travel company has ditched its data warehouse with a data lakehouse query engine and is now experiencing 10x better query performance. Read the case study.

• An environmental production company 10xed the cost-effectiveness of its analytical system by switching to a modern open-source data lakehouse query engine. Read the case study.

• An AB testing SAAS platform is unifying its demanding customer-facing workloads on the data lakehouse. Read the case study.

These examples are not unique, and an increasing number of enterprises have already begun making similar changes to finally achieve the flexibility, scalability, and cost-effectiveness that they had adopted a data lake for in the first place.

**

The Next Step**

To experience and reap the benefits as industry leaders have, consider trying CelerData Cloud Serverless for free. This move could be the best decision for your engineers, analysts, and overall business, paving the way for unmatched flexibility, scalability, and cost-effectiveness.

If you’re interested in the StarRocks project, have questions, or simply seek to discover solutions or best practices, join our StarRocks community on Slack. It’s a great place to connect with project experts and peers from your industry. You can also visit the StarRocks forum for more information.

Sources of Information

We’ve consulted StarRocks committers (Heng Zhao, StarRocks TSC member; Dorian Zheng, StarRocks Active Contributor). As far as Trino, we used the Trino website and google search to research various topics. We compared the latest release of both products as of Oct 2023.

The Rise of Trino/Presto

Initially, Presto was conceived and developed at Facebook (now known as Meta) to enable their data analysts to execute interactive queries on their extensive Apache Hadoop data warehouse. The project, spearheaded by Martin Traverso, Dain Sundstrom, David Phillips, and Eric Hwang, began in 2012 as a solution to overcome the limitations of Apache Hive, which was previously used for SQL analytics on Facebook’s voluminous data warehouse but was found too sluggish for the company’s expansive data needs. Presto was publicly deployed at Facebook in the same year and later made open source in November 2013.

In 2013, when it came out, it had some major advantages.

• Could handle large datasets and complex queries efficiently (compared to other available technology at the time).
  a. Specifically, much faster than MapReduce technology like Apache Hive, which was the incumbent at the time.
  b. Could connect to many different data sources: specifically, connect to multiple databases of the same type with the option to join datasets across databases (e.g. horizontal scaling of database instances).

• Could scale to meet the needs of large organizations: Facebook proved Presto could work, and other tech unicorns quickly adopted it for their data warehouse needs.

• Open Source: who doesn’t like someone else to do R&D and software engineering for a low cost of “free”?

The Presto project underwent significant changes over a decade. In 2018–2019, following the departure of the original founders from Facebook, the project split into two forks: PrestoDB and PrestoSQL. This division was a response to the evolving needs and direction of the Presto community.

Trino emerged from the PrestoSQL fork. In January 2021, PrestoSQL was rebranded as Trino. Trino maintained its roots in large-scale data processing, adopting a Massively Parallel Processing (MPP) architecture and being developed in Java. This distinguished it from traditional map-reduce frameworks, enhancing its ability to efficiently handle and process large data volumes.

The Change in the Users’ Requirement

Since the advent of Trino/Presto, they have satisfactorily met most users’ needs in data analysis at the time. However, it is noteworthy that users’ requirements for data analysis are still constantly changing and evolving. This is particularly evident after the world has been dominated by mobile internet and SaaS applications, with User-facing Analytics and Real-time Analytics becoming important trends for enterprises.

The main manifestations of this trend are as follows:

• Enterprises hope to have more high-performance query engines to meet the demand for low-latency queries on massive amounts of data. No user wants to wait more than three seconds in front of a screen.

• Enterprises need the capacity to support hundreds, or even thousands, of people conducting data queries and analyses simultaneously. The continuously increasing number of users has spurred this demand.

• Enterprises aim to achieve timely analysis of the latest data and use the analysis results to guide subsequent work.

• In the current post-pandemic era, how to save costs and improve work efficiency in such an adverse economic environment? That is another question every enterprise needs to answer.

It is precisely because of these new trends that several database engineers initiated a new database project in 2020, named StarRocks, and officially opened its source code in September 2021. StarRocks was donated to the Linux Foundation at the beginning of 2023. Although it hasn’t been established for long, the influence of StarRocks seems to be growing rapidly. Currently, hundreds of large enterprises around the world are using StarRocks in production environments.

Looking at the use cases, StarRocks and Trino/Presto have a considerable degree of overlap. Simply put, StarRocks is more suited for user-facing scenarios with low latency, while Trino/Presto is more suitable for analytical scenarios that involve fetching data from multiple data sources simultaneously.

Similarities Between Trino and StarRocks

StarRocks and Trino have many similarities in terms of technical features.

Massively Parallel Processing (MPP)

Both engines adopt MPP as their distributed execution framework. In this framework, a query request is split into numerous logic and physical execution units and runs simultaneously on multiple nodes. Unlike the scatter-gather pattern used by many other data analytics products in their distributed computing framework, MPP can utilize more resources to process query requests. Because of this framework, both engines can be used on petabytes of data, and hundreds of giants have already used these engines in their production environments.

Cost-based Optimizer (CBO)

Both engines have Cost-based Optimizer. In multi-table join queries, in addition to the execution engine, optimized execution plans can also play an essential role in improving query performance. Because of the CBO, both engines can support a variety of SQL features, including complex queries, joins, and aggregations. Both Trino and StarRocks have passed TPC-H and the more difficult TPC-DS benchmark.

Pipeline Execution Framework

Both engines have Pipeline execution framework. The primary goal of the Pipeline execution framework is to enhance the efficiency of how a query engine utilizes multi-core resources on a single machine. Its main functions encompass three aspects:

• Reduce the cost of task scheduling for various computing nodes in the query engine.

• Increase CPU utilization while processing query requests.

• Automatically adjust the parallelism of queries execution to fully leverage the computational power of multi-core systems, thereby enhancing query performance.

ANSI SQL Support

Both engines are ANSI SQL compliant. That means that analysts can use the query language that they are most familiar with for their daily work without the need for additional learning costs. The BI tools that enterprises often use will also integrate very easily with StarRocks or Trino.

Differences Between Trino and StarRocks

Although there are some similarities in technical implementation, we can also see some clearly different technical characteristics between these two kinds of systems.

Vectorized Query Engine

StarRocks is a Native Vectorized Engine implemented in C++, while Trino is implemented in Java and uses limited vectorization technology. Vectorization technology helps StarRocks utilize CPU processing power more efficiently. This type of query engine has the following characteristics:

• It can fully utilize the efficiency of columnar data management. This type of query engine reads data from columnar storage, and the way they manage data in memory, as well as the way operators process data, is columnar. Such engines can use the CPU cache more effectively, improving CPU execution efficiency.

• It can fully utilize the SIMD instructions supported by the CPU. This allows the CPU to complete more data calculations in fewer clock cycles. According to data provided by StarRocks, using vectorized instructions can improve overall performance by 3–10 times.

• It can compress data more efficiently to greatly reduce memory usage. This makes this type of query engine more capable of handling large data volume query requests.

In fact, Trino is also exploring vectorization technology. Trino has some SIMD code, but it’s behind compared to StarRocks in terms of depth and coverage. Trino is still working on improving their vectorization efforts (read https://github.com/trinodb/trino/issues/14237). Meta’s Velox project aims to use vectorization technology to accelerate Trino queries. However, so far, very few companies have formally used Velox in production environments.

Materialized View

StarRocks has several materialized view features that Trino does not have. The materialized view is an advanced way to accelerate common queries. Both StarRocks and Trino support creating materialized views however, StarRocks has the ability to

• Automatically re-write queries to enhance query performance. That means StarRocks automatically selects suitable materialized views to accelerate queries. The users don’t need to rewrite their SQLs to make use of the materialized views.

• Execute partition-level materialized view refresh, which allows the user to have better performance and scalability while reducing resource consumption.

• Have the option of writing materialized views to the local disk instead of back to remote disk/storage. That means the users can leverage the high performance of the local disk. Local storage utilizes StarRocks’ proprietary columnar storage format, which better supports the execution of the vectorized query engine.

Trino currently doesn’t have these features:

• It doesn’t have automatic query rewrite features. The user needs to spend a lot of time on query rewriting.

• It needs to execute full-table materialized view refreshes when the data is changed.

• It can not write materialized views on the local disk.

There is an ongoing discussion on how to improve the materialized view “fresh-ness”.

Cache System

The cache system in StarRocks is more complicated than Trino’s. StarRocks implements a cluster-aware data cache on each node. This cache utilizes a combination of memory and disk that can be used for intermediate and final query results. As a result of this component, StarRocks has the ability to cache Apache Iceberg metadata on local disks for better query performance. StarRocks also support warming up the cache, setting cache priorities, and setting cache blacklists.

StarRocks’ query cache significantly enhances query performance in high-concurrency scenarios. It functions by caching the intermediate results of each computing node in memory for subsequent reuse. Query cache is different from the conventional result cache. While the result cache is effective only for identical queries, the query cache can also accelerate queries that are not exactly the same. According to tests by StarRocks development engineers, the query cache can improve query efficiency by 3 to 17 times.

Trino’s cache system is only on the memory level. This makes it very fast, and puts on a need for more numerous and larger memory virtual machine instances. There is work to support local disk caching for “hot cache”.

Read more at https://github.com/trinodb/trino/pull/16375 and https://github.com/trinodb/trino/pull/18719

Join Performance

Both Trino and StarRocks can support complex Join operations. However, StarRocks is capable of delivering higher performance. This is because, in addition to a vectorized query engine, StarRocks also possesses some special technical capabilities.

Join reordering is a technique that can be used to improve the performance of database queries that involve multiple joins. It works by changing the order in which the joins are executed.

The cost of executing a join query depends on the size of the tables being joined and the order in which the joins are executed. By reordering the joins, it is possible to find a join plan that is more efficient. Join reordering can be performed by the optimizer, or it can be specified manually by the user. The optimizer will typically try to reorder the joins to minimize the cost of the query.

There are a number of different algorithms that can be used to reorder joins. Some of the most common algorithms implemented by StarRocks include:

• Greedy algorithm: The greedy algorithm works by repeatedly selecting the pair of tables that has the lowest join cost and joining them together.

• Dynamic programming algorithm: The dynamic programming algorithm works by building a table that contains the cost of joining each pair of tables. The algorithm then uses this table to find the optimal join plan.

• Exhaust algorithm: A technique for performing data joins that is particularly well-suited for large datasets. It works by breaking down the join operation into smaller, more manageable tasks. This makes it possible to perform joins on datasets that are too large to fit in memory.

• Left-deep join reordering: A heuristic algorithm used to optimize the order of joins in a query. The algorithm works by recursively building a left-deep join tree, where each node in the tree represents a join operation. The algorithm starts with the smallest table and then recursively joins it with the next largest table, until all of the tables have been joined.

• Join Associativity algorithm: A technique for optimizing the order of joins in a query. It works by exploiting the associativity property of joins, which states that the order of joins can be changed without affecting the result.

• Join Commutativity algorithm: A technique for optimizing the order of joins in a query. It works by exploiting the commutativity property of joins, which states that the order of join operands can be changed without affecting the result.

Overall, StarRocks implements (at last count) 5 more algorithms than Trino.

Another major feature of StarRocks for Join performance is the runtime filter. Runtime filtering is a technique that can be used to improve the performance of data join operations. It works by filtering out rows from one table before they are joined with another table, based on the join condition. This can significantly reduce the amount of data that needs to be processed, which can lead to significant performance improvements.

• Support Local And Global Runtime Filter

• Shuffle Aware

• Push down Max/Min, In Filter To Storage Engine

• Cost Estimation Based

• Support Runtime Filter Cache

• Push Runtime Filter To Two Sides

• SIMD Bloom Filter

• Adaptive Join Runtime Filters Selection

• Support multi column runtime filter

Finally, StarRocks can support co-located join. A co-located join is a type of join in which the tables being joined are stored on the same nodes of a distributed database cluster. This can significantly improve the performance of the join operation, as the data does not need to be transferred between nodes to be processed.

High Availability

StarRocks has two types of nodes, each capable of achieving high availability through specific strategies. Front End nodes are stateless, and high availability can be achieved by deploying an odd number of Front End nodes. These nodes use the Raft protocol for leader election among themselves. Back End nodes support a multi-replica mechanism, ensuring that the failure of any node does not affect the system’s operation. Therefore, StarRocks can implement hot upgrades of the system. During system upgrades, the online services of the system will not be affected.

Trino does not have built-in high availability (HA) support. Trino’s coordinator is a single point of failure in the system. If this node fails, the entire system becomes unavailable. This means that whenever the system is upgraded, Trino’s online services need to be halted for a period of time. So far, the Trino project has not offered a solution to this issue. Read more at https://github.com/trinodb/trino/issues/391.

Data Sources and Open Table Formats

As advocates of the Data Mesh concept, the Trino community has always been dedicated to integrating more data sources. So far, Trino has developed over 60 different connectors, enabling connections to various data sources, including relational databases, data lakes, and others. This allows Trino to act as a unified query engine for enterprises, facilitating joint analysis of data from different sources. This is especially useful for larger enterprises with multiple businesses and diverse data sources. Currently, StarRocks is more focused on querying Open Data Lakes and has fewer connectors for other data sources.

StarRocks supports read for both Apache Iceberg, Apache Hudi, Apache Hive, and Delta Lake. StarRocks also supports limited writing abilities on Apache Iceberg. Benchmark testing shows that StarRocks is faster as a query engine for data lakes. Trino supports reading and writing for both Apache Iceberg, Apache Hudi, Apache Hive, and Delta Lake. According to StarRocks’ roadmap, the ability to write on the open data lakes will be enhanced soon.

StarRocks’ Data Lakehouse Capability

It is precisely because of these unique technical features that StarRocks can provide users with a more complete Lakehouse experience. Utilizing StarRocks to directly query data lakes can achieve performance comparable to that of data warehouses. This enables many business applications to be built directly on data lakes, eliminating the need to import data into data warehouses for analysis. StarRocks’ Cache system can use the local storage of computing nodes to cache data, transparently accelerating query performance. Users do not need to build additional Pipelines to manage data transfer.

In some user-facing data analysis scenarios, where lower query latency and higher query concurrency are required, StarRocks’ materialized views play a significant role. Materialized views not only speed up related queries by utilizing the local storage of computing nodes, but also their data updates are automatic, requiring no manual intervention. Furthermore, the auto-rewrite feature of materialized views allows users to enjoy the accelerated effects of the views without rewriting any SQL.

Through the combination of various unique technologies, StarRocks truly achieves a user-friendly and high-performance open-source Lakehouse.

Benchmark

The StarRocks team did a benchmark test on the TPC-DS 1TB dataset. They used StarRocks and Trino to query the same copy of data that is stored in Apache Iceberg table format with Parquet files. The result is that Trino’s overall query response time is 5.54x slower than that of StarRocks. See more at https://www.starrocks.io/blog/benchmark-test

Conclusion

Trino/Presto is a very famous open-source query engine. When enterprises have multiple data sources and need to analyze data from these sources in a unified manner, Trino is an appropriate choice. Compared to Trino, StarRocks is an emerging open-source query engine with many innovative and unique designs. Using StarRocks as a query engine for data lakes, customers can easily achieve a high-performance querying experience. Moreover, customers can use various methods to further accelerate queries, achieving lower latency and higher concurrency. StarRocks is also an excellent choice for querying data lakes.

Join Us on Slack

If you’re interested in the StarRocks project, have questions, or simply seek to discover solutions or best practices, join our StarRocks community on Slack. It’s a great place to connect with project experts and peers from your industry. You can also visit the StarRocks forum for more information.

Originally published at https://www.starrocks.io/blog/comparison-of-the-open-source-query-engines-trino-and-starrocks