Vector by Datadog is a high-performance, end-to-end (agent & aggregator) observability data pipeline that lets you collect, transform, and route all your logs and metrics. Additionally, it is open source. While it can serve as an aggregator itself, one can find it more effective to use in conjunction with a specialized data storage tool, such as . Vector.dev Manticore Let’s look at how they can work together. For this, we’ll use an example of indexing , a standard log file of the Debian package manager. dpkg.log The log itself has a simple structure, as shown below: 2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none> Configuration Here is an example of the Vector.dev’s configuration file in the toml format: [sources.test_file]
type = "file"
include = [ "/var/log/dpkg.log" ]

[transforms.modify_test_file]
type = "remap"
inputs = [ "test_file" ]
source = """
.vec_timestamp = del(.timestamp)"""

[sinks.manticore]
type = "elasticsearch"
inputs = [ "modify_test_file" ]
endpoints = ["http://127.0.0.1:9308"]
bulk.index = "dpkg_log" Note that, in this example, we assume Manticore uses its default HTTP port 9308. If you use a custom http port, you should change your Vector.dev config appropriately. Also note that we added the section to the config to rename the default field as it’s a reserved word in Manticore. transforms timestamp Results Now just start with the config above, and the data from the dpkg log will be passed to Manticore and properly indexed. Here is the resulting schema of the created table and an example of the inserted document: Vector.dev mysql> DESCRIBE dpkg_log;
+-----------------+---------+--------------------+
| Field           | Type    | Properties         |
+-----------------+---------+--------------------+
| id              | bigint  |                    |
| file            | text    | indexed stored     |
| host            | text    | indexed stored     |
| message         | text    | indexed stored     |
| source_type     | text    | indexed stored     |
| vec_timestamp   | text    | indexed stored     |
+-----------------+---------+--------------------+


mysql> SELECT * FROM testlog_3 LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353672195
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 startup archives install
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203091741Z
*************************** 2. row ***************************
id: 7856533729353672196
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203808861Z
*************************** 3. row ***************************
id: 7856533729353672197
file: /var/log/dpkg.log
host: logstash-787f68f6f-nhdd2
message: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47
source_type: file
vec_timestamp: 2023-08-04T15:32:50.203814031Z Conclusion Thus, with the integration outlined in this guide, you can now easily and effectively index your log data by employing Manticore in collaboration with Vector by Datadog, a high-performance end-to-end observability data pipeline. This synergy between Vector.dev and Manticore not only offers a streamlined approach for managing log data but also extends the functionality by allowing transformations and routing. Whether you are dealing with simple or complex log structures, this integration provides a robust solution, making the process of collecting, transforming, and storing your data more accessible and efficient. Also published here.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

This writer has a vested interest be it monetary, business, or otherwise, with 1 or more of the products or companies mentioned within.

Integrating Manticore with Vector.dev

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Principles of Proper Database Benchmarking

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

03/09/2018: Biggest Stories in the Cryptosphere

The Noonification: Immigrant Teens Are Working Dangerous Night Shifts in Factories (11/21/2022)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

10 Ways to Optimize Your Database

10 Principles of Proper Database Benchmarking

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

03/09/2018: Biggest Stories in the Cryptosphere

The Noonification: Immigrant Teens Are Working Dangerous Night Shifts in Factories (11/21/2022)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

10 Ways to Optimize Your Database

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps