The above figure shows Node-2 and Node-3 Collectors collect data and input to Node-1 where the database server is installed.
- In Node-2 and Node-3, the Collectors can be seen as running a separate process on a specific log file to send data.
- You can see that each collector process gets detailed information about the log data using a given tpl file .
- The Collector manager is installed in each Node, manages and monitors the Collector process running on that node.
The main features of the Machbase Collector are described below.
Machbase does not require any additional programs in addition to SQL-based commands to execute the Collector. Simply use the following command to manage and monitor the Collector.
Improved Data Collection Performance
Machbase Collector is designed to collect data with a separate Collector for each log data type so each process can process each log file at high speed.
Since separate processes each process log data, they are not affected by other log file processing.
Collector is executed with optimized code for each log type, and data is input with dedicated protocol that minimizes resource usage, so that the best performance can be obtained.
Collectors can be used to collect log data in a variety of ways. The data collection method can be set by modifying the tpl file. The following collection methods are supported.
Collects files from local host.
Collects files from remote host.
Collects data coming into port.
Collects files from other databases.
Log Data Types
Machbase Collector supports regular expressions for various types of log data.
The user can simply modify the existing regular expression to analyze various log files. Currently, the following log types are supported.
Regular Expression File Name
Data Default Location (can be modified)
Machbase trace log
Apache web server access file
Apache web server error file
Easily Supports Custom Logs
Machbase Collector can process various kinds of log files that can be represented as regular expressions.
Even if you do not have a log file, you can test sample log messages and regular expressions using machregex.
Prevents Data Loss in the Event of a Failure
Machbase Collector provides the ability to correctly retransmit data that failed to be transmitted in the event of a failure, after the failure has been resolved.
When a failure occurs, the Collector records the last location it sent to the server, resets the fault, and then resends the data from that location.
So, even if you do not write any additional operations or code to overcome the obstacle, you can transfer it to the server without losing any data.
Ensures High Availability
To ensure high availability of services, multiple Collectors can operate simultaneously on the same data source, and these Collectors transfer data to different Machbase servers.
In this way, even if an error occurs in the Machbase server, the same data is continuously stored in another server, so that the service can be continued.
After resolving the error and restarting the server, the Collector can retransmit the untransmitted log data correctly, thus automatically replicating the data to provide high availability.
Integrated Monitoring Through MWA
Machbase Collector manager synchronizes the Collector's execution information to the Machbase server.
Using this, it is possible to perform integrated monitoring through MWA (Machbase Web Admin).
By using MWA, it is possible to monitor various status information of the running Collector and the status information of the server running the Collector in real time.
Log Pre-Processing Using Python Script
You can write a Python script to manipulate the Collector before it processes the data.
The input data can be processed so as not to input unnecessary data, or the parsed data can be changed.