Log rotation demo with Systemd Unit

-
System Units
To demonstrate log rotation using logrotate and helps to understand systemD unit creation and apply compute resoruce(memory/CPU) to specific cgroup.

  • Create script that generates logs

/home/scripts/script_logrotate.sh
#!/bin/bash
# https://redis.io/docs/latest/develop/use/patterns/bulk-loading/
# Set the number of keys to generate
num_keys=2000

# Set the size of the value for each key (in bytes)
# 1MB , with k and v, -> 2MB
value_size=64

# create logs dir
mkdir -p /home/scripts/logs/

# Set the output file name
output_file="/home/scripts/logs/redis-data.log"

# Generate data and write to file
for i in $(seq 1 $num_keys); do
  # Generate a random string of the specified size
  value=$(head -c $value_size /dev/urandom | tr -dc A-Za-z0-9 | head -c $value_size)
  # Create the Redis SET command
  redis_command1="SET key$i \"$value\""

  # Append the command to the output file
 echo "$redis_command1" >> $output_file

done

echo "Redis data generation complete. Data written to $output_file"

System Units

  • Create System Unit file and place under /etc/systemd/system/logrotate.service

/etc/systemd/system/logrotate.service
[Unit]
Description=LogRotateService
After=network.target

[Service]
Slice=tvajjala.slice
Type=simple
ExecStart=/bin/bash /home/scripts/script_logrotate.sh
Restart=always

# 1/4 th of CPU core
CPUAccounting=yes
CPUWeight=1000
# run on CPU core 0
CPUAffinity=0

CPUQuota=100
# max threads it can open
TasksAccounting=yes
TasksMax=5

PassEnvironment=arch=%a
# allowing for monitoring and potentially limiting memory usage.
# maximum memory used by process
MemoryAccounting=yes
MemoryMax=20M

OOMScoreAdjust=1000

[Install]
WantedBy=multi-user.target
you can paas environment variables using PassEnvironment=arch-%a. refer more about available variable https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html

  • Start service using below commands

commands.sh
systemctl status logrotate.service

systemctl start logrotate.service

With above steps logs are gets generated under /home/scripts/logs

Log Rotation

Log rotation is an automated system administration process where log files are archived, compressed, renamed, or deleted based on size or time, preventing disk space overflow and ensuring manageable log data.

  • Create Log rotation configuration file default config file available at /etc/logrotate.conf

/home/config/logrotate-demo.conf
/home/opc/scripts/logs/*.log {

    # dont send email
    nomail

    # size <size>: Rotates based on a specified size
    size 500M

    #hourly
    hourly

    # missingok: Specifies that logrotate should skip missing log files without error.
    missingok

    # rotate <count>: Determines how many rotated log files to keep before deleting the oldest ones
    rotate 1

    #notifempty: Ensures that empty log files are not rotated.
    notifempty

    # safer option for applications that hold a lock on the log file
    copytruncate

    # which user doing log rotation. usually root on instance.
    # if you running  inside root less containers use same user that you pass keep-id /userid inside container
    su root root
    # if app expect log use this option provide necessary access
    # create new (empty) log files after rotating old ones
    # inside the container user is games , keep same permission
    create 644 games games

    # use date as a suffix of the rotated file
    dateext

    # uncomment this if you want your log files compressed
    #compress

    # RPM packages drop log rotation information into this directory
    include /etc/logrotate.d
}

  • Run script using below command , default state file located at /var/lib/logrotate/

run.sh
# syntax : logrotate -vs state-file config-file
/usr/sbin/logrotate  /home/opc/scripts/logrotate.conf --state /home/opc/scripts/logrotate.custom.state --verbose

# using logrotate.crontab.
# NOTE: state file is optional if not provided it creates at `/var/lib/logrotate/`
# TIP: https://crontab.guru/#*/60_*_*_*_*
*/5 * * * * /usr/sbin/logrotate  /home/opc/scripts/logrotate.conf

#
crontab /home/logrotate.crontab

Reference

Comments

Post a Comment

Popular posts from this blog

IBM Datapower GatewayScript

-
Introduction IBM API Assembly Language Introduction GatewayScript is a new transformation technology for API, Web, and Mobile and is available in DataPower. GatewayScript is based on the JavaScript programming language. More precisely, it supports the ECMAScript 5.1 version of the specification. On the DataPower appliance, JavaScript runs in ‘strict’ mode only. The strict mode provides a more consistent and more disciplined behavior. For example, if a script attempts to write a property of a DataPower API object and that object is frozen, an immediate exception is thrown. The ‘strict’ mode behavior supports stronger error checking and handling and increases security. DataPower Playground (GatewayScript Fiddle) IBM DataPower Playground is an interactive website that lets you write GatewayScript code and execute it on a cloud-hosted DataPower Gateway for learning purposes. Available at : https://datapower-playground.mybluemix.net/ GatewayScript provides ...
Read more

Spring boot SOAP Web Service Performance

-
Image
Consumng This tutorial walks you through the process of consuming a SOAP-based web service with Spring. It elucidates following items Making parallel calls with RxJava (or WebFlux ) Sending large SOAP message payloads using AXIOM( AXis Object Model) AxiomSoapMessageFactory .(which improves performance) Marshalling( is the process of transforming the memory representation of an object to a data format suitable for storage or transmission) Marshalling (XML Serialization) done with O/X Mappers, different types of mappers out there and choose appropriate one for your need. Castor XML mapping is an open source XML binding framework. It allows you to transform the data contained in a java object model into/from an XML document. By default, it does not require any further configuration, though a mapping file can be used to have more control over the behavior of Castor. The Spring integration classes reside in the org.springframework.oxm.castor pa...
Read more

Jenkins Basics

-
Jenkins Build Parameters Table of Contents Jenkins Build Parameters Invoking Downstream Job with Build Parameters Reading Job results from child Job Global Tools Wait until block Edit properties/ json file within Jenkins Jenkins Job kill forcibly How to push Code to Git repository from Jenkins How to run parallel stages Jacoco Integration in build.gradle Jacoco PublishReport Plugin Publishing Artifacts and HTMLReport to Build Job Jenkins HTML Display Error Reset Jenkins Job Number Jenkins is the widely adopted open source continuous integration tool. You can orchestrate any application deployments using Jenkins with a wide range of plugins and native Jenkins workflows. In This Tutorial we will be demonstrates interesting features of jenkins pipeline. In this tutorial declarative pipeline syntax will be used. Note official documentation for jenkins pipeline syntax Jenkins Build Parameters Build Parameters makes jenkins job more d...
Read more