Machine Learning Build system: Using Qwak

In the previous blog post, we demonstrated how to implement an ML build system. We retrieved the training data, trained a classification model, and created a REST service in a Docker image. In the end, we started the Docker container, ran the tests, and uploaded the image to ECR. 

The code we wrote was quite verbose because we needed to write the Dockerfile, install all required libraries, and copy the serialized model. Also, the only way to reuse the code is by copying and pasting parts of it. To deploy a new model, we’ll need to repeat the entire process. Imagine how difficult it would become if we had a dozen models running in production.

Thankfully, we can simplify model training and deployment by using Qwak. Qwak handles model training, dockerization, testing, and deployment---all from the Qwak platform.

In this example, we'll train a Titanic classifier using CatBoost.

Project setup

To start, we have to create a Qwak project and a model within the project using the Qwak command-line tool:

 qwak projects create --project-name the_project_name --project-description "Put the description here"

The command will return the project UUID, which we’ll need later to create the model using the command-line interface (CLI). 

Alternatively, we could create the project using the web UI. In the Projects view, click the “Create New Project” button and provide the project name and the description:

‍

From here, we  create a new model. First, let’s do it using the CLI. Note that the command creates only a model placeholder in the Qwak platform.

 qwak models create --project-id project_uuid --model-name the_model_name --model-description "Model description"

We can also use the web interface to create a model. Let’s open the project created in the previous step and click the “Create New Model” button in the project view:

Now we need to prepare the project directory. For that, we will use the Qwak CLI:

qwak models init directory_name

The command-line tool will ask us about the model directory name and the model class name. When we specify the required information, it will create two directories: main and tests.

In the main directory, we need three files: __init__.py, conda.yml, and model.py. We will use them to configure the runtime environment and build the model.

In the conda.yml file, we put the dependencies required during training, runtime, and testing. 

name: the_project_name
channels:  
- defaults  
- conda-forge
dependencies:  
- python=3.8  
- pip=19.3.1  
- pandas=1.1.5  
- scikit-learn=0.24.1  
- catboost=0.26.1

Training the model

In the model.py file, we extend the QwakModelInterface class. In the class, we put the training code and the inference code. Let's start with the imports:

import numpy as np
import pandas as pd
import qwak

from catboost import CatBoostClassifier, Pool, cv
from catboost.datasets import titanic
from qwak.model.base import QwakModelInterface
from qwak.model.schema import ExplicitFeature, ModelSchema, Prediction
from sklearn.model_selection import train_test_split

In the constructor, we will prepare the CatBoostClassifier classifier. We need to put the model in an object field because we need access to the same model.

class TitanicSurvivalPrediction(QwakModelInterface):
	def __init__(self):
		loss_function='Logloss'
		learning_rate=None
		iterations=1000
		custom_loss='Accuracy'‍

		self.model = CatBoostClassifier(iterations=iterations,
			custom_loss=[custom_loss],
  		loss_function=loss_function,
  		learning_rate=learning_rate)

Let’s say we would like to log the training parameters. Qwak gives us a specialized method for logging training parameters, which we can use in the constructor:

qwak.log_param({    
'loss_function' : loss_function,    
'learning_rate' : learning_rate,    
'iterations' : iterations,    
'custom_loss' : custom_loss
})

Now we can start training the model. We'll need to extend the build method. In the method body, we'll put the same code as in the previous article:

def build(self):    
titanic_train, _ = titanic()    
titanic_train.fillna(-999, inplace=True)‍    

x = titanic_train.drop(['Survived','PassengerId'], axis=1)    
y = titanic_train.Survived‍    

x_train, x_test, y_train, y_test = train_test_split(x, y, train_size=.85, random_state=42)‍    

cate_features_index = np.where(x_train.dtypes != float)[0]‍    

self.model.fit(x_train, y_train, cat_features=cate_features_index, eval_set=(x_test, y_test))‍    

cv_data = cv(Pool(x, y, cat_features=cate_features_index), self.model.get_params(), fold_count=5)

At the end of the build method, we can also log the accuracy of the trained model using the Qwak API:

qwak.log_metric({"val_accuracy" : np.max(cv_data["test-Accuracy-mean"])})

That's enough to train a model in the Qwak platform, but before we initiate it, we must also prepare the inference code in the TitanicSurvivalPrediction class.

Inference code

First, we specify the schema used by the REST API. The schema is optional, but if we define it, Qwak will show us example inference code in the Qwak UI:

We'll define the input schema and the schema of the returned JSON object. Then, we'll pass the entire row from the Titanic dataset. In our API, we also give the PassengerId, which is useless for the classification. However, we'll do it anyway to demonstrate the data preprocessing in a Qwak model.

def schema(self):    
	return ModelSchema(        
		features=[            
    ExplicitFeature(name='PassengerId', type=int),            
    ExplicitFeature(name='Pclass', type=int),            
    ExplicitFeature(name='Name', type=str),            
    ExplicitFeature(name='Sex', type=str),            
    ExplicitFeature(name='Age', type=int),            
    ExplicitFeature(name='SibSp', type=int),            
    ExplicitFeature(name='Parch', type=int),            
    ExplicitFeature(name='Ticket', type=str),            
    ExplicitFeature(name='Fare', type=float),            
    ExplicitFeature(name='Cabin', type=str),            
    ExplicitFeature(name='Embarked', type=str),        
   ],        
   predictions=[            
   	Prediction(name='Survived_Probability', type=float)        
   ])

In the end, we need to implement the inference method. This method receives a pandas DataFrame with the input data, executes the data preprocessing, obtains a prediction from the model, and returns the response in the same format as the one defined in the schema:

@qwak.analytics()
def predict(self, df: pd.DataFrame) -> pd.DataFrame:    
	df = df.drop(['PassengerId'], axis=1)    
	return pd.DataFrame(self.model.predict_proba(df)[:, 1], columns=['Survived_Probability'])

Note the `@qwak.analytics()` decorator. The decorator instructs the Qwak platform to track the requests and metadata related to runtime performance. Later, we can check the request elapsed time and memory usage, or retrieve the request logs.

The last thing we must do is create the model object in the __init__.py file:

from .model import TitanicSurvivalPrediction‍

def load_model():    
	return TitanicSurvivalPrediction()

Building the model

We can use the Qwak platform to train the model:

qwak models build --model-id "model_identifier" path_to_the_code_directory

The Qwak command-line tool will upload the code to the server, install the dependencies, and train the model. We'll see the training result in the Qwak web UI. In the UI, we can also deploy a trained model in the Qwak platform and use the Python API to infer the classification.

However, before we retrieve the predictions, we will add the testing code to the project.

Testing the model

As a part of the model build, we can run test cases to verify whether the model works correctly. In the `tests/it` directory, we can create a test class test_titanic.py that uses the Qwak mock client to verify the model predictions:

import pandas as pd
from qwak_mock import real_time_client‍

def test_realtime_api(real_time_client):    
	feature_vector = [        
  	{            
    	'PassengerId': 762,            
      'Pclass': 3,            
      'Name': "Nirva, Mr. Iisakki Antino Aijo ",            
      'Sex': "female",            
      'Age': 34,            
      'SibSp': 4,            
      'Parch': 3,            
      'Ticket': "a",            
      'Fare': 1.0,            
      'Cabin': "A",            
      'Embarked': "A"        
     }]‍    
     
     survived_probability: pd.DataFrame = real_time_client.predict(feature_vector)    
     assert survived_probability['Survived_Probability'].values[0] > 0

Obtaining predictions from a deployed model

After building the model and deploying it, we can use the Qwak inference client to obtain a prediction from the model running in the Qwak platform:

from qwak.inference.clients import RealTimeClient‍

feature_vector = [    
	{        
  	'PassengerId': 762,        
    'Pclass': 3,        
    'Name': "Nirva, Mr. Iisakki Antino Aijo ",        
    'Sex': "female",        
    'Age': 34,        
    'SibSp': 4,        
    'Parch': 3,        
    'Ticket': "a",        
    'Fare': 1.0,        
    'Cabin': "A",        
    'Embarked': "A"    
   }]‍
   
 client = RealTimeClient(model_id="the_model_identifier")
 client.predict(feature_vector)
 

Qwak

Using Qwak, we minimized the training, testing, and serialization mode. Contrary to what we did in the previous article, we write only the model-specific code and don't need any boilerplate.