This research presents a novel system designed for continuous analysis of enormous data using stream processing. The system consists of several modules including metadata management, query plan generation, data import task generation, increment processing, MR message processing, and database connection. The metadata management module handles the management of meta-information for data tables and databases. The query plan generation module receives query requests and generates optimized query plans. The data import task generation module handles data import requests and generates data import MR operation sets. The increment processing module incrementally processes data import and query operations in parallel using a Hadoop system. The MR message processing module receives results from Map or Reduce functions of the Hadoop system and outputs them to the next operation or the Reduce end. Finally, the database connection module serves as an interface between the Hadoop system and the databases. The proposed system leverages the Hadoop system to organize databases in a distributed manner and allows simultaneous execution of data import and query operations. Additionally, the system incorporates a pipeline technology to enhance the MR execution flow, enabling continuous stream mode execution of data queries and significantly reducing the time required for analyzing enormous data.

Stream processing, Continuous analysis, Enormous data, Metadata management, Query plan generation, Data import, MR message processing, Database connection.