A software architecture for monitoring big data storage platforms

Abstract

NoSQL-based big data storage platforms provide similar fundamental big data management functionalities in addition to various other functionalities that differ for each platform. For example, document oriented NoSQL big data storage platforms are commonly used for organizing and managing documents, while graph-based databases are designed for data whose relations are represented by graphs. In turn, different NoSQL-based platforms are often used together, as each provides distinct capabilities. Within this study based on our literature review, it is seen that a hybrid platform, which could perform real-time monitoring tasks on top of different big data NoSQL platforms, is lacking. In order to address this issue, this paper proposes a novel system architecture. The proposed system architecture runs as a piece of add-on software one layer above the NoSQL platforms and provides monitoring tasks on these platforms. Within the research, a prototype of the recommended system architecture is made and the testing results are provided in detail. The prototype of the proposed architecture focused on monitoring the system's distributed structure and data structure, memory usage, and disk usage. In order to prove the practical usage of the proposed system architecture, various performance experiments were applied to the prototype application. In this paper, we report on the promising results of the performance experiment.