Pull data from Snowflake with logstash

Izek Chen
1 min readFeb 10, 2020

--

Note how the config looks like.

We are looking for a solution to display snowflake to Grafana. I know that have a schedule to pull the data out of the data lake looks strange, but with no choice. Unfortunately, using Grafana to pull data directly from snowflake is expensive. Therefore we need an alternative. Below is the data flow:

snowflake ← logstash → ES

Here is the logstash JDBC input example
logstash.yml with snowflake JDBC connector

input {
jdbc {
jdbc_driver_library => "/etc/logstash/pipeline/snowflake/dependency/snowflake-jdbc-3.9.2.jar"
jdbc_driver_class => "com.snowflake.client.jdbc.SnowflakeDriver"
jdbc_connection_string => "jdbc:snowflake://url.eu-west-1.snowflakecomputing.com/?db=dbname&warehouse=House"
jdbc_user => "user"
jdbc_password => "password"
schedule => "15 * * * *"
statement => "
SELECT TOP 100000 * FROM dbname.CORE.table WHERE timestamp >= '2038-10-02 00:00:00' AND timestamp < '2038-10-02 23:59:59'
"
jdbc_fetch_size => 1000
}
}

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

--

--

Izek Chen
Izek Chen

Written by Izek Chen

Technical Account manager @ Google Cloud

Responses (3)

Write a response