Example (1) Creating Regular Expression/Template File
This example is to collect data in SOCKET mode. SOCKET mode opens a socket and waits for an external program to enter data.
It is recommended to use rsyslog to collect syslogs. This example uses rsyslog to collect and analyze data.
Data transmission is performed by another program. In the following configuration file, only the configuration of the rgx file and the tpl file is handled.
The syslog.rgx used in the FILE mode data processing example is still available.
In the example below, the socket port number is 33333. This port must allow access from the firewall.
Example (2) Running Collector
Use the socket_syslog.tpl file in the example above to create a collector and run it. (Creation and execution are the same as FILE mode.)
Run the following command to verify that the collector is running normally and is connected.
From the above results, you can see that the collector is waiting for data input on port 33333.
Example (3) Data Insert
Enter the data using rsyslog.
Log in the rsyslog program as a user with root privileges and create the rsyslog configuration file (/etc/rsyslog.d/.conf) as follows.
Or you can make the following detailed settings.
Create the following file as "/etc/rsyslog.d/127.0.0.1_syslog.conf" file. When rsyslog is restarted, it outputs data to socket every time syslog data is generated.
Then restart the rsyslog daemon.
When rsyslog is restarted, /var/log/syslog data is written to the port 127.0.0.1:33333.
The collector gathers data through a socket connection, analyzes it, and enters it into the Machbase server.
Since the $InputFilePollInterval variable is set to 1 in the rsyslog setting, the data input rate may be slow at the beginning.
If the data is entered normally, the database server can check the data in the corresponding table using the following SELECT query.
Log Collector Configuration
The socket input mode example is executed using rsyslog, logstash, and nxlog .
When these programs input log data through a socket, the collector collects them and inputs them to the database server.
rsyslog is often included by default in recent Linux distributions.
So there is no need to install it any more. Just add the configuration file to the /etc/rsyslog.d/ directory and restart the rsyslog daemon.
rsyslog does not only transmit log data that is already recorded, but also transmits data whenever a new log data is recorded.
Below is a list of settings.
Simple Setting: Set Log Forwarding Address
This is the only way to specify the address to be forwarded when the log is created. It is simpler than other methods.
Restart rsyslog after writing the following in /etc/rsyslog.d/.conf file.
The syslog data is then sent to collector host: port .
Complex Setting: Set Input Log File / Transmission Frequency
A more complex method is to set the input log file and transmission frequency.
Create a file that ends in .conf in the /etc/rsyslog.d/ folder, and then restart rsyslog.
See the rsyslog description for more details .
To install logstash, refer to Getting Started with Logstash .
You can modify the logstash conf file to send the desired data to the socket.
See the example below.
- Set the location of the input data file in the "input" section. If you want to enter syslog, set /var/log/syslog.
- In the "output" section, you need to enter the collector's tcp socket, so set tcp and set ip and port number.
Nxlog is a log collector for Windows.
The configuration of the rgx, tpl file for the collector for the socket input mode is the same, and an example of the configuration file for nxlog is as follows.
Usually nxlog is installed in "C:\Program Files\nxlog" or "C:\Program Files (x86)\nxlog".
Create a configuration file located in the above path as follows.
In the example above, when data is written to the im_msvistalog file, data transmission through the socket is set to <collector ip>: <collector port>.
Changing the configuration file and restarting the service will send the data to the collector via the socket.
Refer to the nxlog manual for details.
ODBC mode is a method of collecting data from a database that can be accessed by an ODBC connection to a collector.
In Linux environments, you need to install the unixODBC package.
The following example shows how to collect data from a MySQL database through unixODBC.
Please refer to the respective websites for how to install unixODBC and MyODBC .
Additional Value Configuration
The following variables must be set.
DSN for accessing the database
You must use the DSN described in the ODBC configuration.
Query for data retrieval
Column name of incremental value
Must be one of the columns to be queried. Only numeric types are supported.
Example (1) Generating Data
You must first enter the MySQL data. Enter data as follows.
The table columns to be collected are seq, at, srcip, srcport, dstip, dstport, protocol, eventlog, eventcode, and eventsize. Set seq as a sequential increment column. To create a table with this structure, perform the following query in mysql:
After successfully creating the table, enter the data into the MySQL database with the following command:
For more information about running MySQL databases, see the MySQL manual.
Example (2) ODBC Configuration
Create an ODBC configuration file to access the MySQL database.
In unixODBC, USER DATA SOURCES is used first, so make the above contents in the .odbc.ini file in the user home directory that runs collector.
Example (3) Checking ODBC Settings
Use the unixODBC isql program to verify that the ODBC configuration is working properly. Execute the following as a parameter of MYSQL which is the set DSN.
If it is installed and configured normally, you will get the following results.
You can query the input data through isql.
If you can not get these results, check the names of unixODBC and DSN. Refer to the unixODBC documentation for related details.
Example (4) Collector Configuration
Create a tpl file using the query, DSN, username and password used above.
In the ODBC mode input method, REGEX_PATH is not needed because data is provided separately for each column.
Use the above tpl file to create and start the collector, and then check the results using machsql.
You can see that the data is output in reverse order to the MySQL output.