TencentDB TDStore Online DDL: Technological Evolution and Innovations Background & Challenges
SHENZHEN, China, April 22, 2025 /PRNewswire/ — In the evolving world of data management, traditional single-node databases like MySQL rely on OnlineDDL and third-party tools such as pt-online-schema-change (pt-osc) to facilitate schema changes without locks. These methods, while effective, face significant performance challenges and become cumbersome in distributed environments. Addressing these very challenges is Tencent Cloud’s TDStore, a cutting-edge, financial-grade distributed database, which introduces remarkable technological advancements.
Core Technological Innovations
1. Multi-Version Schema Mechanism
The multi-version schema mechanism is a standout innovation. By introducing schema versioning, TDStore enables metadata-only modifications to occur in mere seconds, such as adding trailing columns or extending fields. This system smartly ensures that historical data auto-populates with default values, maintaining seamless backward compatibility and enhancing user experience.
2. Concurrency Control & State Transition
With an emphasis on refining concurrency control, TDStore integrates two essential mechanisms:
a. Thomas Write Rule: This cleverly minimizes transaction conflicts by simply disregarding outdated, stale writes. The result is a noticeable improvement in DDL-DML parallel operations.
b. Google F1 Phased State Design: Inspired by Google’s approach, the system divides DDL processes into a tri-phased transition: delete-only, write-only, and final. This design ensures not only global consistency but also smooth and reliable transitions.
3. Write Fence Mechanism
The write fence mechanism shines in its ability to validate request versions directly at the storage layer. By permitting writes only between consecutive states, it effectively mitigates data inconsistency risks, ensuring data integrity.
4. Fast OnlineDDL Acceleration
TDStore’s OnlineDDL acceleration is nothing short of revolutionary. Utilizing distributed parallel backfilling, it segments data into SST files which are then ingested across multiple nodes through bulk load. This bypasses traditional timestamp comparisons, producing a staggering 13-fold performance improvement – reducing schema change times from 2.3 hours to just 10 minutes.
Practices & Optimizations
1. Performance Comparison
The results of implementing Fast OnlineDDL are compelling:
Traditional Mode (single-node): Utilizing 16 threads, schema changes previously required 2.3 hours.
Fast Mode (multi-node): By employing 48 threads, the process now concludes in a mere 10 minutes, underscoring significant efficiency gains and time savings for users.
2. Partitioning Best Practices
Efficiency in handling large data sets is critical, and TDStore provides innovative partitioning techniques:
a. Large Tables: Implementing HASH/KEY partitioning effectively distributes data, facilitating parallel DDL operations.
b. Cold/Hot Separation: The combination of RANGE and HASH secondary partitioning enables swift data cleanup and flexible scaling.
c. High Concurrency: Aligning partition keys with frequently queried fields and setting partition counts as multiples of node numbers ensures optimal performance.
3. Key Parameter Configuration
Configuration options allow for further customization and enhancement:
a. max_parallel_ddl_degree: This parameter allows users to increase parallel threads, up to the total number of CPUs within nodes.
b. tdsql_ddl_fillback_mode: Enabling IngestBehind mode unlocks the potential for multi-node parallel acceleration, vastly increasing performance.
Business Value & Future Roadmap
Tencent Cloud’s TDStore’s capabilities have been validated across numerous real-world applications, achieving zero downtime in petabyte-scale financial systems and executing up to 10 times faster than conventional third-party tools.
The future looks promising with upcoming enhancements aimed at optimizing partitioned table operations and index backfilling for ordinary tables. Furthermore, it is set to support ultra-large-scale workloads, enhancing its suitability for hybrid HTAP architectures.
Conclusion
TDStore stands out by overcoming the limitations of traditional OnlineDDL approaches through its distributed architecture innovations and state-of-the-art engineering practices. It delivers robust, secure, and high-performance schema change capabilities tailored for demanding financial-grade scenarios. Enterprises can now tackle massive data management challenges with confidence, backed by Tencent Cloud’s pioneering technology.
This technological leap epitomizes how distributed databases are pivotal for the future of data management, ensuring enterprises remain agile and competitive.
Stay up-to-date with the latest advancements in distributed database technology, empowering your organization to excel in the digital era.
#DistributedDatabase #TencentCloud #TencentDB #TDSQL #TencentCloudBigData
