The various data acquisition modes supported by the Machbase collector and how to apply them are described here. Collector supports FILE, SFTP, SOCKET, and ODBC data collection modes.

The description below assumes that the collector and the Machbase server are installed on the same machine, the hostname of the server is "localhost", and the collector is running at 127.0.0.1:9999.

Data Input Mode Configuration

The data insertion mode can be changed by adjusting the COLLECT_TYPE variable. 

You can now configure FILE, SFTP, SOCKET, ODBC, and so on. Additional variables must be set for each mode.

Index


FILE Method


When the file mode input is set, the collector reads and processes the file of the running server.

Additional Option Configuration

In file mode, data is read from localhost, so you only need to check the pathname and file read permissions of that file.

Option Name

Description

Remarks

LOG_SOURCE

Path where log file is located

Must be written in absolute path.


Example (1) Checking Log File


The following example is the file mode input method that collects data from the "/var/log/syslog" file and inputs it to the Machbase server.
First, you need to make sure that the input file can be read by the collector process.

[mach@localhost ~]$ head /var/log/syslog
head: cannot open '/var/log/syslog' for reading: Permission denied


The above results show that there is no read access to the input file.
You need to grant read permission to the input file (/var/log/syslog) to the user running collector so that the collector process can read the input file.
The following example shows the process.

[mach@localhost ~]$ su
# chmod 744 /var/log/syslog
# exit
[mach@localhost ~]$


Again, if you check whether you can read the input file using the head command, you get the following result.

[mach@localhost ~]$ head /var/log/syslog
Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start
Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
[mach@localhost ~]$


Example (2) Creating Regular Expression File

After verifying the permissions on the input file, you need to create a regular expression file to parse the data.

You can use the machregex tool provided by Machbase to check that your regular expression matches the data format. 

The machregex tool sets two regular expressions as parameters and enters data.

  • The first regular expression (REGEX) is used to parse the input data.
  • The second regular expression (END_REGEX) is used to separate input data one by one. 

The following example shows how to run machregex using regular expressions and data files.

[mach@localhost ~]$ head /var/log/syslog > syslog
[mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog
Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$)
============================================================================
SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.
)
  ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.
]
  0 (0:15) => [Jun 20 04:31:43]
  1 (0:3) => [Jun]
  2 (4:6) => [20]
  3 (7:15) => [04:31:43]
  4 (16:25) => [localhost]
  5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started.
]

The output value of the parsing result is represented by six tokens. According to this file, Machbase uses 0, 4, and 5 tokens and discards the remaining tokens.

  • 0: Time representation string. To convert this string to datetime, you must specify format.
  • 4: Host name.
  • 5: Log data message.

If the data you want to analyze is processed without any problems according to the above parsing rules, you can use the same syslog.rgx file provided by Machbase. 

(This file is in the $MACHBASE_HOME/collector/regex/ folder.)
If you use the regular expression file in this folder, you can set only the file name without using the file path in the REGEX_PATH variable of the template file .


Example (3) Creating Template File

Below is an example of a syslog.tpl template file.

###############################################################################
# Copyright of this product 2013-2023,
# Machbase Inc. or its subsidiaries.
# All Rights reserved
###############################################################################

#
#  This file is for Machbase collector template file.
#

###################################################################
# Collect setting
###################################################################

COLLECT_TYPE=FILE

LOG_SOURCE=/var/log/syslog

###################################################################
# Process setting
###################################################################

REGEX_PATH=syslog.rgx

###################################################################
# Output setting
###################################################################

DB_TABLE_NAME = "file_syslogtable"
DB_ADDR       = "127.0.0.1"
DB_PORT       = 5656
DB_USER       = "SYS"
DB_PASS       = "MANAGER"

# 0: Direct insert
# 1: Prepared insert
# 2: Append
APPEND_MODE=2

# 0: None, just append.
# 1: Truncate.
# 2: Try to create table. If table already exists, warn it and proceed.
# 3: Drop and create.
CREATE_TABLE_MODE=2

In collector settings, files that are not the default file path must specify an absolute path (a path beginning with '/') and a file name.

The file name to be read is specified in the LOG_SOURCE variable, and the regular expression file for parsing the data must also be set.

Example (4) Running Collector

Set the information for connecting to the Machbase server and the table generation method, and when the template file is set, execute the collector as follows.

Mach> create collector localhost.file_syslog from "/home/machbase_home/collector/syslog.tpl";
Created successfully.

Mach> ALTER COLLECTOR localhost.file_syslog START;
Altered successfully.

If the collector is successfully created and executed (and only if there is no table), the table will be created and data will begin to be input.

To check if the data is being input normally, you can check it by executing a SELECT query on the generated table.

Mach> SELECT * FROM file_syslogtable ORDER BY _arrival_time asc LIMIT 10;
COLLECTOR_TYPE                    COLLECTOR_ADDR  
-----------------------------------------------------
COLLECTOR_ORIGIN                                                                  COLLECTOR_OFFSET     
----------------------------------------------------------------------------------------------------------
TM                              HOST                                                                              
---------------------------------------------------------------------------------------------------------------------
MSG                                                                               
------------------------------------------------------------------------------------
FILE                              127.0.0.1       
/var/log/syslog                                                                   81                   
2016-06-20 04:31:43 000:000:000 localhost                                                                         
kernel: imklog 5.8.10, log source = /proc/kmsg started.                           
FILE                              127.0.0.1       
/var/log/syslog                                                                   217                  
2016-06-20 04:31:43 000:000:000 localhost                                                                         
rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h  
ttp://www.rsyslog.com"] start                                                     
FILE                              127.0.0.1       
/var/log/syslog                                                                   256                  
2016-06-20 04:31:46 000:000:000 localhost                                                                         
kernel: imklog 5.8.10, log source = /proc/kmsg started.                                                                      
FILE                              127.0.0.1       
/var/log/syslog                                                                   431                  
2016-06-20 04:35:01 000:000:000 localhost                                                                         
CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
FILE                              127.0.0.1       
/var/log/syslog                                                                   606                  
2016-06-20 04:40:01 000:000:000 localhost                                                                         
CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
FILE                              127.0.0.1       
/var/log/syslog                                                                   681                  
2016-06-20 04:40:01 000:000:000 localhost                                                                         
CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1)                                  
FILE                              127.0.0.1       
/var/log/syslog                                                                   856                  
2016-06-20 04:45:01 000:000:000 localhost                                                                         
CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
FILE                              127.0.0.1       
/var/log/syslog                                                                   1031                 
2016-06-20 04:50:01 000:000:000 localhost                                                                         
CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
FILE                              127.0.0.1       
/var/log/syslog                                                                   1106                 
2016-06-20 04:50:01 000:000:000 localhost                                                                         
CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1)                                  
FILE                              127.0.0.1       
/var/log/syslog                                                                   1281                 
2016-06-20 04:55:01 000:000:000 localhost                                                                         
CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
[10] row(s) selected.


SFTP Method


You can use SFTP mode to collect data from remote files.

Remote files must be accessible via SFTP. It is similar to FILE mode, but SFTP related variables should be set because the file is accessed through SFTP.

Additional Option Configuration

To collect data in SFTP mode, the following variables must be set.

Option Name

Description

Remarks

LOG_SOURCE

Data file path and file name of remote location

Must be written in absolute path.

SFTP_HOST

SFTP server IP address


SFTP_PORT

SFTP server port number

Is set to 22 by default if not port number is set.

SFTP_USER

SFTP username


SFTP_PASS

SFTP password



Example (1) Checking SFTP Access

If you can not read the login and collection files with SFTP, you should try the following.

  1. Troubleshoot SFTP sign-in issues 
  2. Troubleshoot file permissions (see FILE method)


Example (2-3) Creating Regular Expression/Template File

Once this is done, create a regular expression file and a template file.
For the description of this operation, refer to FTP method above.

###############################################################################
# Copyright of this product 2013-2023,
# Machbase Corporation (Incorporation) or its subsidiaries.
# All Rights reserved
###############################################################################

#
#  This file is for Machbase collector template file.
#

###################################################################
# Collect setting
###################################################################

COLLECT_TYPE=SFTP

SFTP_HOST=127.0.0.1
SFTP_PORT=22
SFTP_USER=mach
SFTP_PASS=mach

LOG_SOURCE=/var/log/syslog

###################################################################
# Process setting
###################################################################

REGEX_PATH=syslog.rgx

###################################################################
# Output setting
###################################################################

DB_TABLE_NAME = "sftp_syslogtable"
DB_ADDR       = "127.0.0.1"
DB_PORT       = 5656
DB_USER       = "SYS"
DB_PASS       = "MANAGER"

# 0: Direct insert
# 1: Prepared insert
# 2: Append
APPEND_MODE=2

# 0: None, just append
# 1: Truncate
# 2: Try to create table. If table already exists, warn it and proceed.
# 3: Drop and create
CREATE_TABLE_MODE=2


Example (4) Running Collector

The following example shows how to create a collector using SFTP with the above template file.

Mach> create collector localhost.sftp_syslog from "/home/mach/mach_collector_home/collector/sftp_syslog.tpl";
Created successfully.

Mach> alter collector localhost.sftp_syslog start;
Altered successfully.

If you have successfully created and started the Collector, you can see the data collected by the Collector as follows:

Mach> select * from sftp_syslogtable order by _arrival_time asc limit 10;
COLLECTOR_TYPE                    COLLECTOR_ADDR  
-----------------------------------------------------
COLLECTOR_ORIGIN                                                                  COLLECTOR_OFFSET     
----------------------------------------------------------------------------------------------------------
TM                              HOST                                                                              
---------------------------------------------------------------------------------------------------------------------
MSG                                                                               
------------------------------------------------------------------------------------
SFTP                              127.0.0.1       
/var/log/syslog                                                                   81                   
2016-06-20 04:31:43 000:000:000 localhost                                                                         
kernel: imklog 5.8.10, log source = /proc/kmsg started.                           
SFTP                              127.0.0.1       
/var/log/syslog                                                                   217                  
2016-06-20 04:31:43 000:000:000 localhost                                                                         
rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h  
ttp://www.rsyslog.com"] start                                                     
SFTP                              127.0.0.1       
/var/log/syslog                                                                   256                  
2016-06-20 04:31:46 000:000:000 localhost                                                                         
kernel: imklog 5.8.10, log source = /proc/kmsg started.                                                                      
SFTP                              127.0.0.1       
/var/log/syslog                                                                   431                  
2016-06-20 04:35:01 000:000:000 localhost                                                                         
CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
SFTP                              127.0.0.1       
/var/log/syslog                                                                   606                  
2016-06-20 04:40:01 000:000:000 localhost                                                                         
CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
SFTP                              127.0.0.1       
/var/log/syslog                                                                   681                  
2016-06-20 04:40:01 000:000:000 localhost                                                                         
CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1)                                  
SFTP                              127.0.0.1       
/var/log/syslog                                                                   856                  
2016-06-20 04:45:01 000:000:000 localhost                                                                         
CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
SFTP                              127.0.0.1       
/var/log/syslog                                                                   1031                 
2016-06-20 04:50:01 000:000:000 localhost                                                                         
CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
SFTP                              127.0.0.1       
/var/log/syslog                                                                   1106                 
2016-06-20 04:50:01 000:000:000 localhost                                                                         
CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1)                                  
SFTP                              127.0.0.1       
/var/log/syslog                                                                   1281                 
2016-06-20 04:55:01 000:000:000 localhost                                                                         
CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc  
k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)              
[10] row(s) selected.
  • No labels