Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Machbase Collector is a tool that extracts log data and inputs the data to the Machbase database in real time after converting it.

Machbase Collector can collect log data in real time and input it through the network by being installed in a separate device from the Machbase server. It operates as a separate process from the Machbase server and can run multiple Collectors at the same time. Each Collector process processes one data source.

Concept



UI Text Box
목차

Index


Table of Contents
maxLevel3
indent30px
exclude목차
classtoc


The above figure shows Node-2 and Node-3 Collectors collect data and input to Node-1 where the database server is installed.

  • In Node-2 and Node-3, the Collectors can be seen as running a separate process on a specific log fileto send data. 
  • You can see that each collector process gets  detailed information about the log data using a given tpl file .
  • The Collector manager  is installed in each Node, manages and monitors the Collector process running on that node.


Characteristic


The main features of the Machbase Collector are described below.

Consistent Interface

Machbase does not require any additional programs in addition to SQL-based commands to execute the Collector. Simply use the following command to manage and monitor the Collector.

Code Block
languagesql
CREATE Collector MANAGER LOCALHOST AT '127.0.0.1:9999';
CREATE Collector LOCALHOST.MYADP FROM 'syslog.tpl';
ALTER Collector LOCALHOST.MYADP START;

Improved Data Collection Performance

Machbase Collector is designed to collect data with a separate Collector for each log data type so each process can process each log file at high speed.

Since separate processes each process log data, they are not affected by other log file processing. 
Collector is executed with optimized code for each log type, and data is input with dedicated protocol that minimizes resource usage, so that the best performance can be obtained.


Collection Method

Collectors can be used to collect log data in a variety of ways. The data collection method can be set by modifying the tpl file. The following collection methods are supported.

Method Name

Description

FILE

Collects files from local host.

SFTP

Collects files from remote host.

SOCKET

Collects data coming into port.

ODBC

Collects files from other databases.

Log Data Types

Machbase Collector supports regular expressions for various types of log data. 

The user can simply modify the existing regular expression to analyze various log files. Currently, the following log types are supported.

Regular Expression File Name

Supported Type

Data Default Location (can be modified)

machbase.rgx

Machbase trace log

$MACHBASE_HOME/trc/machbase.trc
apache_access.rgx

Apache web server access file

/var/log/apache2/access.log
apache_error.rgx

Apache web server error file

/var/log/apache2/access.log
syslog.rgx

sysglog 파일sysglog file

/var/log/syslog
custom.rgx

유저 정의 타입

Custome type

유저 정의 파일

Custom file

Easily Supports Custom Logs

Machbase Collector can process various kinds of log files that can be represented as regular expressions.

Even if you do not have a log file, you can test sample log messages and regular expressions using machregex. 

Prevents Data Loss in the Event of a Failure

Machbase Collector provides the ability to correctly retransmit data that failed to be transmitted in the event of a failure, after the failure has been resolved.

When a failure occurs, the Collector records the last location it sent to the server, resets the fault, and then resends the data from that location.
So, even if you do not write any additional operations or code to overcome the obstacle, you can transfer it to the server without losing any data.


Ensures High Availability

To ensure high availability of services, multiple Collectors can operate simultaneously on the same data source, and these Collectors transfer data to different Machbase servers.

In this way, even if an error occurs in the Machbase server, the same data is continuously stored in another server, so that the service can be continued.
After resolving the error and restarting the server, the Collector can retransmit the untransmitted log data correctly, thus automatically replicating the data to provide high availability.

MWA를 통한 통합 모니터링

Integrated Monitoring Through MWA

마크베이스 Collector manager는 Collector 의 실행 정보를 마크베이스 서버에 동기화 한다.
이를 이용하여 MWA (Machbase Web Admin) 를 통한 통합 모니터링을 수행할 수 있다.
MWA를 이용하면, 실행 중인 Collector 의 다양한 상태 정보와 Collector 를 수행 중인 서버의 상태 정보를 실시간으로 모니터링할 수 있다.
Machbase Collector manager synchronizes the Collector's execution information to the Machbase server.

Using this, it is possible to perform integrated monitoring through MWA (Machbase Web Admin).
By using MWA, it is possible to monitor various status information of the running Collector and the status information of the server running the Collector in real time.

Python 스크립트를 이용한 로그 전처리

Log Pre-Processing Using Python Script

Python 스크립트를 작성하여 Collector 가 데이터를 처리하기 전에 조작할 수 있다.
입력한 데이터를 처리하여 필요 없는 데이터는 입력하지 않도록 설정할 수도 있고 파싱한 데이터를 변경하도록 할 수도 있다.

You can write a Python script to manipulate the Collector before it processes the data.

The input data can be processed so as not to input unnecessary data, or the parsed data can be changed.