Pull data from Snowflake with logstash

Izek Chen
1 min readFeb 10, 2020

--

Note how the config looks like.

We are looking for a solution to display snowflake to Grafana. I know that have a schedule to pull the data out of the data lake looks strange, but with no choice. Unfortunately, using Grafana to pull data directly from snowflake is expensive. Therefore we need an alternative. Below is the data flow:

snowflake ← logstash → ES

Here is the logstash JDBC input example
logstash.yml with snowflake JDBC connector

input {
jdbc {
jdbc_driver_library => "/etc/logstash/pipeline/snowflake/dependency/snowflake-jdbc-3.9.2.jar"
jdbc_driver_class => "com.snowflake.client.jdbc.SnowflakeDriver"
jdbc_connection_string => "jdbc:snowflake://url.eu-west-1.snowflakecomputing.com/?db=dbname&warehouse=House"
jdbc_user => "user"
jdbc_password => "password"
schedule => "15 * * * *"
statement => "
SELECT TOP 100000 * FROM dbname.CORE.table WHERE timestamp >= '2038-10-02 00:00:00' AND timestamp < '2038-10-02 23:59:59'
"
jdbc_fetch_size => 1000
}
}

--

--

Izek Chen
Izek Chen

Written by Izek Chen

Technical Account manager @ Google Cloud

Responses (3)