Back to blog
Announcing Qwak Feature Store support for Snowflake data sources
Announcements

Announcing Qwak Feature Store support for Snowflake data sources

By 
Pavel Klushin
September 27, 2021

Announcing Qwak Feature Store support for Snowflake data sources


Qwak Feature Store provides a unified feature store for training and real-time operations without the need to write additional code or create manual processes to keep features consistent.  There are many ways to create features in the Qwak Feature Store, including Batch-based features, streaming or non-materialized features.


Starting today, Qwak supports Snowflake as a data source for Batch features


The Data Source connectors provide you with a consistent data source interface for any database and create a standard way to combine stream and batch data sources for Feature Transformations.

How it’s done


Defining a Feature Set enables you to create features from your analytical data: when calculating feature values, Qwak will simply read from the underlying data source.


For example, in a fraud detection model use case, we might have two values to pull from a Snowflake data source:

  • Average transaction per customer - avg_amount
  • Standard deviation of a transaction per customer - sttdev_amount

And two from Streaming events from Kafka:

  • Last transaction amount
  • Last transaction time


Architecture



How to configure Snowflake as Data Source


Snowflake data source connector definition:



from qwak.feature_store.sources.data_sources import SnowflakeSource

some_snowflake_source = SnowflakeSource(name='users_demo',
                                        description='a snowflake source description',
                                        date_created_column='insert_date_column',
                                        host='Snowflake DNS address',
                                        username_secret_name='snowflake-username',
                                        password_secret_name='snowflake-password',
                                        database='db_name',
                                        schema='schema_name',
                                        warehouse='data_warehouse_name',
                                        sql='SQL query for feature extraction')


Register batch feature using the Snowflake connector:



BatchFeatureSet(
  name=”batch_transaction_features”,
  data_sources=[“snowflake_transactions_history”],
  scheduling_policy=”daily”,
  validations=[expect_column_values_to_be_between(
    column=”amount”, min_value=0, max_value=None)]
  function=SqlFunction(
    “””
    SELECT User_ID,
      avg(Amount) AS avg_amount,
      sttdev(Amount) AS stddev_amount
    FROM snowflake_trasactions_history
    GROUP BY User_ID
    “””)
)



Qwak Feature Store helps ensure models make accurate predictions by making the same features available for both training and inference.

 

For more information, feel free to reach out at support@qwak.ai


Related articles