[an error occurred while processing this directive]

Coal Engineering ›› 2026, Vol. 58 ›› Issue (7): 145-151.doi: 10.11799/ce202607018

Previous Articles     Next Articles

Data transmission and storage for multi-source mine group data based on message queue#br#

  

  • Received:2025-09-22 Revised:2025-11-04 Online:2026-07-15 Published:2026-08-03

Abstract:

To address issues such as poor real-time performance in multi-source data transmission and the lack of breakpoint resume mechanisms in intelligent coal mining, this paper proposes a message queue-based distributed data collaborative transmission and storage scheme. By constructing a hybrid message queue architecture integrating Kafka and RabbitMQ, the scheme decouples data sending and receiving. It enhances concurrent transmission capability by combining an event-driven non-blocking I/O model with a multiplexing mechanism, and designs an intelligent breakpoint resume module to ensure data integrity.The overall architecture of the scheme consists of four layers: data sources, centralized control acquisition, data transmission, and data monitoring platform. Equipment data is uniformly collected through industrial protocols such as OPC UA, encrypted, and then distributed to Kafka topics. A ClickHouse distributed cluster is utilized to achieve PB-level data storage.Experimental data show that in applications across more than ten typical working faces, the scheme accumulatively collects over 1 billion data records in a single month without any loss, with data transmission delay reduced to the second level. Its efficiency is significantly improved compared with traditional methods, providing reliable technical support for data management and control of mine groups in coal mining enterprises.

CLC Number: