Recently in my practice, I faced a significant challenge: extracting application logs per day from Loki in the Kubernetes environment. When cold, the application writes about 60 lines per minute, and when someone interacts with the application, it can write 2000-5000 lines of logs per minute – it turns out that was necessary to get more than 300,000 lines of logs. The setup did not include a configured log export, and the primary log viewing tool was Grafana, which imposes a 5000-line limit on log retrieval. Increasing this limit was not feasible as it would significantly strain our resources and was unnecessary for this one-time task. Additionally, accessing logs directly from the Kubernetes pod was not an option due to storage limitations within the pod itself. So, I need to download logs directly from Loki without changing configurations. Preparation Used tools LogCli
Kubectl Additional steps To ensure that the query will use to search for logs is correct, follow these steps: Navigate to Grafana explore:

Go to Grafana > Explore


Set the required label:

Apply the necessary label to filter logs by the service.


Filter by date:


Use the operation filter to display lines containing the desired date. Example query: {instance="our-service"} |= `2024-07-12` Execution Install LogCli:

Download the LogCli binary from the Loki releases page



Set Loki address:

Configure the Loki address for LogCli using an environment variable:

export LOKI_ADDR=http://localhost:8000



Port forwarding:

Forward local ports to the Loki pod to allow local access:

kubectl --namespace loki port-forward svc/loki-stack 8000:3100



Extract logs:

Use LogCli to query and save the logs to a file:

logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log


In this command:


--limit is set with a high value to ensure all logs are captured.


--since is set to 72 hours to cover a sufficient time range. Conclusion This entire process took approximately 10 minutes, resulting in a file with the complete application logs for the specified date. If needed, this process can be further optimized or automated. Recently in my practice, I faced a significant challenge: extracting application logs per day from Loki in the Kubernetes environment. When cold, the application writes about 60 lines per minute, and when someone interacts with the application, it can write 2000-5000 lines of logs per minute – it turns out that was necessary to get more than 300,000 lines of logs. The setup did not include a configured log export, and the primary log viewing tool was Grafana, which imposes a 5000-line limit on log retrieval. Increasing this limit was not feasible as it would significantly strain our resources and was unnecessary for this one-time task. Additionally, accessing logs directly from the Kubernetes pod was not an option due to storage limitations within the pod itself. So, I need to download logs directly from Loki without changing configurations. Preparation Used tools LogCli Kubectl LogCli LogCli LogCli Kubectl Kubectl Kubectl Additional steps To ensure that the query will use to search for logs is correct, follow these steps: Navigate to Grafana explore:

Go to Grafana > Explore Set the required label:

Apply the necessary label to filter logs by the service. Filter by date:


Use the operation filter to display lines containing the desired date. Navigate to Grafana explore: Go to Grafana > Explore Go to Grafana > Explore Go to Grafana > Explore Grafana Explore Set the required label: Apply the necessary label to filter logs by the service. Apply the necessary label to filter logs by the service. Apply the necessary label to filter logs by the service. label Filter by date: Use the operation filter to display lines containing the desired date. Use the operation filter to display lines containing the desired date. Use the operation filter to display lines containing the desired date. Use the operation filter to display lines containing the desired date. operation Example query: Example query: {instance="our-service"} |= `2024-07-12` {instance="our-service"} |= `2024-07-12` Execution Install LogCli:

Download the LogCli binary from the Loki releases page Set Loki address:

Configure the Loki address for LogCli using an environment variable:

export LOKI_ADDR=http://localhost:8000 Port forwarding:

Forward local ports to the Loki pod to allow local access:

kubectl --namespace loki port-forward svc/loki-stack 8000:3100 Extract logs:

Use LogCli to query and save the logs to a file:

logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log


In this command:


--limit is set with a high value to ensure all logs are captured.


--since is set to 72 hours to cover a sufficient time range. Install LogCli: Download the LogCli binary from the Loki releases page Install LogCli: Download the LogCli binary from the Loki releases page Download the LogCli binary from the Loki releases page Loki releases page Loki releases page Set Loki address: Configure the Loki address for LogCli using an environment variable: export LOKI_ADDR=http://localhost:8000 Set Loki address: Configure the Loki address for LogCli using an environment variable: Configure the Loki address for LogCli using an environment variable: export LOKI_ADDR=http://localhost:8000 export LOKI_ADDR=http://localhost:8000 Port forwarding: Forward local ports to the Loki pod to allow local access: kubectl --namespace loki port-forward svc/loki-stack 8000:3100 Port forwarding: Forward local ports to the Loki pod to allow local access: Forward local ports to the Loki pod to allow local access: kubectl --namespace loki port-forward svc/loki-stack 8000:3100 kubectl --namespace loki port-forward svc/loki-stack 8000:3100 Extract logs: Use LogCli to query and save the logs to a file: logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log In this command:


--limit is set with a high value to ensure all logs are captured.


--since is set to 72 hours to cover a sufficient time range. Extract logs: Use LogCli to query and save the logs to a file: Use LogCli to query and save the logs to a file: logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log logcli query '{instance="our-service"} |= `2024-07-12`' --limit=5000000 --since=72h -o raw > our-service-2024-07-12.log In this command:


--limit is set with a high value to ensure all logs are captured.


--since is set to 72 hours to cover a sufficient time range. In this command: --limit is set with a high value to ensure all logs are captured.


--since is set to 72 hours to cover a sufficient time range. --limit is set with a high value to ensure all logs are captured. --since is set to 72 hours to cover a sufficient time range. --limit is set with a high value to ensure all logs are captured. --limit is set with a high value to ensure all logs are captured. --since is set to 72 hours to cover a sufficient time range. --since is set to 72 hours to cover a sufficient time range. Conclusion This entire process took approximately 10 minutes, resulting in a file with the complete application logs for the specified date. If needed, this process can be further optimized or automated.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Fetching Large Logs from Loki in Kubernetes

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Accessing Private Google Artifact Registry with Poetry: Local and Docker Setup

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

Accessing Private Google Artifact Registry with Poetry: Local and Docker Setup

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps