煤炭工程 ›› 2026, Vol. 58 ›› Issue (7): 145-151.doi: 10.11799/ce202607018

• 研究探讨 • 上一篇    下一篇

基于消息队列的多源矿群数据传输与存储技术研究

唐 博,西成峰,王 辉,吴早阳   

  1. 北京天玛智控科技股份有限公司,北京 101300
  • 收稿日期:2025-09-22 修回日期:2025-11-04 出版日期:2026-07-15 发布日期:2026-08-03
  • 通讯作者: 唐博 E-mail:18800192889@163.com

Data transmission and storage for multi-source mine group data based on message queue#br#

  • Received:2025-09-22 Revised:2025-11-04 Online:2026-07-15 Published:2026-08-03

摘要:

针对煤矿智能化开采中多源数据传输实时性差、断点续传机制缺失等问题,提出一种基于消息队列的分布式数据协同传输与存储方案。通过构建KafkaRabbitMQ混合消息队列架构,实现数据发送与接收解耦,结合事件驱动非阻塞I/O模型与多路复用机制提升并发传输能力,并设计智能断点续传模块保障数据完整性。方案整体架构包含数据源、集控采集、数据传输和数据监测平台四层,通过OPC UA等工业协议统一采集设备数据,经加密处理后分发至Kafka主题,利用ClickHouse分布式集群实现PB级数据存储。试验数据表明,该方案在十余个典型工作面应用中,单月累计采集数据超10亿条且无丢失,数据传输延迟缩短至秒级,较传统方法效率提升显著,为煤矿集团矿群数据管控提供了可靠的技术支撑。

Abstract:

To address issues such as poor real-time performance in multi-source data transmission and the lack of breakpoint resume mechanisms in intelligent coal mining, this paper proposes a message queue-based distributed data collaborative transmission and storage scheme. By constructing a hybrid message queue architecture integrating Kafka and RabbitMQ, the scheme decouples data sending and receiving. It enhances concurrent transmission capability by combining an event-driven non-blocking I/O model with a multiplexing mechanism, and designs an intelligent breakpoint resume module to ensure data integrity.The overall architecture of the scheme consists of four layers: data sources, centralized control acquisition, data transmission, and data monitoring platform. Equipment data is uniformly collected through industrial protocols such as OPC UA, encrypted, and then distributed to Kafka topics. A ClickHouse distributed cluster is utilized to achieve PB-level data storage.Experimental data show that in applications across more than ten typical working faces, the scheme accumulatively collects over 1 billion data records in a single month without any loss, with data transmission delay reduced to the second level. Its efficiency is significantly improved compared with traditional methods, providing reliable technical support for data management and control of mine groups in coal mining enterprises.

中图分类号: