Use python script to edit data in logstash

I have a logstash configuration that gets data from a MySQL database and sends the data to elasticsearch. This is my configuration:

input {
  jdbc {
    clean_run => true
    jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://IP:PORT/DATABASE"
    jdbc_user => "myuser"
    jdbc_password => "mypassword"
    use_column_value => true
    tracking_column => "field1"
    schedule => "*/2 * * * *"
    statement => "SELECT * FROM test"
  }
}

**need something here before the 'output' section ?**

output {
  elasticsearch {
    hosts => ["http://ELASTICSEARCH_IP:PORT"]
    index => "myindexname"
    document_id => "%{field1}"
  }
}

Everything’s working fine, but I need to add some columns that have values dependent upon other column values, so I tried writing a Python script to do this. Is there a way to execute a python script to add/edit columns before data are sent in elasticsearch? Do I need the filter option?

EDIT : For exemple, I use my python script to :

create a week number column based on a datetime field.
create a month number column based on a datetime field.
edit the ‘name’ column and replace some special characters (‘/’, ‘-‘, ‘:’, etc…)
create linear trendline based on an another column.
create an moving average line based on an another column.
replace some columns values (ex : replace ‘y’ by ‘yes’ and ‘n’ by ‘no’).

Answer

I finally did the thing with the ruby code. Thanks mates for your help!

Advertisement

Answer