Clipper Connection¶
ClipperConnection
is the primary way of starting and managing a Clipper cluster.
-
class
clipper_admin.
ClipperConnection
(container_manager)[source]¶ -
__init__
(container_manager)[source]¶ Create a new ClipperConnection object.
After creating a
ClipperConnection
instance, you still need to connect to a Clipper cluster. You can connect to an existing cluster by callingclipper_admin.ClipperConnection.connect()
or create a new Clipper cluster withclipper_admin.ClipperConnection.start_clipper()
, which will automatically connect to the cluster once it Clipper has successfully started.Parameters: container_manager ( clipper_admin.container_manager.ContainerManager
) – An instance of a concrete subclass ofContainerManager
.
-
start_clipper
(query_frontend_image='clipper/query_frontend:develop', mgmt_frontend_image='clipper/management_frontend:develop', frontend_exporter_image='clipper/frontend-exporter:develop', cache_size=33554432, qf_http_thread_pool_size=1, qf_http_timeout_request=5, qf_http_timeout_content=300, num_frontend_replicas=1)[source]¶ Start a new Clipper cluster and connect to it.
This command will start a new Clipper instance using the container manager provided when the
ClipperConnection
instance was constructed.Parameters: - query_frontend_image (str(optional)) – The query frontend docker image to use. You can set this argument to specify a custom build of the query frontend, but any customization should maintain API compability and preserve the expected behavior of the system.
- mgmt_frontend_image (str(optional)) – The management frontend docker image to use. You can set this argument to specify a custom build of the management frontend, but any customization should maintain API compability and preserve the expected behavior of the system.
- frontend_exporter_image (str(optional)) – The frontend exporter docker image to use. You can set this argument to specify a custom build of the management frontend, but any customization should maintain API compability and preserve the expected behavior of the system.
- cache_size (int(optional)) – The size of Clipper’s prediction cache in bytes. Default cache size is 32 MiB.
- qf_http_thread_pool_size (int(optional)) – The size of thread pool created in query frontend for http serving.
- qf_http_timeout_request (int(optional)) – The seconds of timeout on request handling in query frontend for http serving..
- qf_http_timeout_content (int(optional)) – The seconds of timeout on content handling in query frontend for http serving..
- num_frontend_replicas (int(optional)) – The number of query frontend to deploy for fault tolerance and high availability.
Raises: clipper.ClipperException
-
register_application
(name, input_type, default_output, slo_micros)[source]¶ Register a new application with Clipper.
An application in Clipper corresponds to a named REST endpoint that can be used to request predictions. This command will attempt to create a new endpoint with the provided name. Application names must be unique. This command will fail if an application with the provided name already exists.
Parameters: - name (str) – The unique name of the application.
- input_type (str) – The type of the request data this endpoint can process. Input type can be one of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
- default_output (str) – The default output for the application. The default output will be returned whenever an application is unable to receive a response from a model within the specified query latency SLO (service level objective). The reason the default output was returned is always provided as part of the prediction response object.
- slo_micros (int) – The query latency objective for the application in microseconds. This is the processing latency between Clipper receiving a request and sending a response. It does not account for network latencies before a request is received or after a response is sent. If Clipper cannot process a query within the latency objective, the default output is returned. Therefore, it is recommended that the SLO not be set aggressively low unless absolutely necessary. 100000 (100ms) is a good starting value, but the optimal latency objective will vary depending on the application.
Raises: clipper.UnconnectedException
clipper.ClipperException
-
link_model_to_app
(app_name, model_name)[source]¶ Routes requests from the specified app to be evaluted by the specified model.
Parameters: Raises: clipper.UnconnectedException
clipper.ClipperException
Note
Both the specified model and application must be registered with Clipper, and they must have the same input type. If the application has previously been linked to a different model, this command will fail.
-
unlink_model_from_app
(app_name, model_name)[source]¶ Prevents the model with model_name from being used by the app with app_name. The model and app should both be registered with Clipper and a link should already exist between them.
Parameters: Raises: clipper.UnconnectedException
clipper.ClipperException
-
build_and_deploy_model
(name, version, input_type, model_data_path, base_image, labels=None, container_registry=None, num_replicas=1, batch_size=-1, pkgs_to_install=None)[source]¶ Build a new model container Docker image with the provided data and deploy it as a model to Clipper.
This method does two things.
1. Builds a new Docker image from the provided base image with the local directory specified by
model_data_path
copied into the image by callingclipper_admin.ClipperConnection.build_model()
.2. Registers and deploys a model with the specified metadata using the newly built image by calling
clipper_admin.ClipperConnection.deploy_model()
.Parameters: - name (str) – The name of the deployed model
- version (str) – The version to assign this model. Versions must be unique on a per-model basis, but may be re-used across different models.
- input_type (str) – The type of the request data this endpoint can process. Input type can be one of “integers”, “floats”, “doubles”, “bytes”, or “strings”. See the User Guide for more details on picking the right input type for your application.
- model_data_path (str) – A path to a local directory. The contents of this directory will be recursively copied into the Docker container.
- base_image (str) – The base Docker image to build the new model image from. This image should contain all code necessary to run a Clipper model container RPC client.
- labels (list(str), optional) – A list of strings annotating the model. These are ignored by Clipper and used purely for user annotations.
- container_registry (str, optional) – The Docker container registry to push the freshly built model to. Note that if you are running Clipper on Kubernetes, this registry must be accesible to the Kubernetes cluster in order to fetch the container from the registry.
- num_replicas (int, optional) – The number of replicas of the model to create. The number of replicas
for a model can be changed at any time with
clipper.ClipperConnection.set_num_replicas()
. - batch_size (int, optional) – The user-defined query batch size for the model. Replicas of the model will attempt to process at most batch_size queries simultaneously. They may process smaller batches if batch_size queries are not immediately available. If the default value of -1 is used, Clipper will adaptively calculate the batch size for individual replicas of this model.
- pkgs_to_install (list (of strings), optional) – A list of the names of packages to install, using pip, in the container. The names must be strings.
Raises: clipper.UnconnectedException
clipper.ClipperException
-
build_model
(name, version, model_data_path, base_image, container_registry=None, pkgs_to_install=None)[source]¶ Build a new model container Docker image with the provided data”
This method builds a new Docker image from the provided base image with the local directory specified by
model_data_path
copied into the image. The Dockerfile that gets generated to build the image is equivalent to the following:FROM <base_image> COPY <model_data_path> /model/
The newly built image is then pushed to the specified container registry. If no container registry is specified, the image will be pushed to the default DockerHub registry. Clipper will tag the newly built image with the tag [<registry>]/<name>:<version>.
This method can be called without being connected to a Clipper cluster.
Parameters: - name (str) – The name of the deployed model.
- version (str) – The version to assign this model. Versions must be unique on a per-model basis, but may be re-used across different models.
- model_data_path (str) – A path to a local directory. The contents of this directory will be recursively copied into the Docker container.
- base_image (str) – The base Docker image to build the new model image from. This image should contain all code necessary to run a Clipper model container RPC client.
- container_registry (str, optional) – The Docker container registry to push the freshly built model to. Note that if you are running Clipper on Kubernetes, this registry must be accesible to the Kubernetes cluster in order to fetch the container from the registry.
- pkgs_to_install (list (of strings), optional) – A list of the names of packages to install, using pip, in the container. The names must be strings.
Returns: The fully specified tag of the newly built image. This will include the container registry if specified.
Return type: Raises: clipper.ClipperException
Note
Both the model name and version must be valid DNS-1123 subdomains. Each must consist of lower case alphanumeric characters, ‘-‘ or ‘.’, and must start and end with an alphanumeric character (e.g. ‘example.com’, regex used for validation is ‘[a-z0-9]([-a-z0-9]*[a-z0-9])?Z’.
-
deploy_model
(name, version, input_type, image, labels=None, num_replicas=1, batch_size=-1)[source]¶ Deploys the model in the provided Docker image to Clipper.
Deploying a model to Clipper does a few things.
1. It starts a set of Docker model containers running the model packaged in the
image
Docker image. The number of containers it will start is dictated by thenum_replicas
argument, but the way that these containers get started depends on your choice ofContainerManager
implementation.2. It registers the model and version with Clipper and sets the current version of the model to this version by internally calling
clipper_admin.ClipperConnection.register_model()
.Notes
If you want to deploy a model in some other way (e.g. a model that cannot run in a Docker container for some reason), you can start the model manually or with an external tool and call
register_model
directly.Parameters: - name (str) – The name of the deployed model
- version (str) – The version to assign this model. Versions must be unique on a per-model basis, but may be re-used across different models.
- input_type (str) –
The type of the request data this endpoint can process. Input type can be one of “integers”, “floats”, “doubles”, “bytes”, or “strings”. See the User Guide for more details on picking the right input type for your application.
- image (str) – The fully specified Docker image to deploy. If using a custom
registry, the registry name must be prepended to the image. For example,
if your Docker image is stored in the quay.io registry, you should specify
the image argument as
“quay.io/my_namespace/image_name:tag”. The image name and tag are independent of
the
name
andversion
arguments, and can be set to whatever you want. - labels (list(str), optional) – A list of strings annotating the model. These are ignored by Clipper and used purely for user annotations.
- num_replicas (int, optional) – The number of replicas of the model to create. The number of replicas
for a model can be changed at any time with
clipper.ClipperConnection.set_num_replicas()
. - batch_size (int, optional) – The user-defined query batch size for the model. Replicas of the model will attempt to process at most batch_size queries simultaneously. They may process smaller batches if batch_size queries are not immediately available. If the default value of -1 is used, Clipper will adaptively calculate the batch size for individual replicas of this model.
Raises: clipper.UnconnectedException
clipper.ClipperException
Note
Both the model name and version must be valid DNS-1123 subdomains. Each must consist of lower case alphanumeric characters, ‘-‘ or ‘.’, and must start and end with an alphanumeric character (e.g. ‘example.com’, regex used for validation is ‘[a-z0-9]([-a-z0-9]*[a-z0-9])?Z’.
-
register_model
(name, version, input_type, image=None, labels=None, batch_size=-1)[source]¶ Registers a new model version with Clipper.
This method does not launch any model containers, it only registers the model description (metadata such as name, version, and input type) with Clipper. A model must be registered with Clipper before it can be linked to an application.
You should rarely have to use this method directly. Using one the Clipper deployer methods in
clipper_admin.deployers
or callingbuild_and_deploy_model
ordeploy_model
will automatically register your model with Clipper.Parameters: - name (str) – The name of the deployed model
- version (str) – The version to assign this model. Versions must be unique on a per-model basis, but may be re-used across different models.
- input_type (str) –
The type of the request data this endpoint can process. Input type can be one of “integers”, “floats”, “doubles”, “bytes”, or “strings”. See the User Guide for more details on picking the right input type for your application.
- image (str, optional) – A docker image name. If provided, the image will be recorded as part of the model descrtipin in Clipper when registering the model but this method will make no attempt to launch any containers with this image.
- labels (list(str), optional) – A list of strings annotating the model. These are ignored by Clipper and used purely for user annotations.
- batch_size (int, optional) – The user-defined query batch size for the model. Replicas of the model will attempt to process at most batch_size queries simultaneously. They may process smaller batches if batch_size queries are not immediately available. If the default value of -1 is used, Clipper will adaptively calculate the batch size for individual replicas of this model.
Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_current_model_version
(name)[source]¶ Get the current model version for the specified model.
Parameters: name (str) – The name of the model
Returns: The current model version
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_num_replicas
(name, version=None)[source]¶ Gets the current number of model container replicas for a model.
Parameters: Returns: The number of active replicas
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
set_num_replicas
(name, num_replicas, version=None)[source]¶ Sets the total number of active replicas for a model.
If there are more than the current number of replicas currently allocated, this will remove replicas. If there are less, this will add replicas.
Parameters: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_all_apps
(verbose=False)[source]¶ Gets information about all applications registered with Clipper.
Parameters: verbose (bool) – If set to False, the returned list contains the apps’ names. If set to True, the list contains application info dictionaries. These dictionaries have the same attribute name-value pairs that were provided to
clipper_admin.ClipperConnection.register_application()
.Returns: Returns a list of information about all apps registered to Clipper. If no apps are registered with Clipper, an empty list is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_app_info
(name)[source]¶ Gets detailed information about a registered application.
Parameters: name (str) – The name of the application to look up Returns: Returns a dictionary with the specified application’s info. This will contain the attribute name-value pairs that were provided to clipper_admin.ClipperConnection.register_application()
. If no application with namename
is registered with Clipper, None is returned.Return type: dict Raises: clipper.UnconnectedException
-
get_linked_models
(app_name)[source]¶ Retrieves the models linked to the specified application.
Parameters: app_name (str) – The name of the application
Returns: Returns a list of the names of models linked to the app. If no models are linked to the specified app, empty list is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_all_models
(verbose=False)[source]¶ Gets information about all models registered with Clipper.
Parameters: verbose (bool) – If set to False, the returned list contains the models’ names. If set to True, the list contains model info dictionaries.
Returns: Returns a list of information about all apps registered to Clipper. If no models are registered with Clipper, an empty list is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_model_info
(name, version)[source]¶ Gets detailed information about a registered model.
Parameters: Returns: Returns a dictionary with the specified model’s info. If no model with name model_name@model_version is registered with Clipper, None is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_all_model_replicas
(verbose=False)[source]¶ Gets information about all model containers registered with Clipper.
Parameters: verbose (bool) – If set to False, the returned list contains the apps’ names. If set to True, the list contains container info dictionaries.
Returns: Returns a list of information about all model containers known to Clipper. If no containers are registered with Clipper, an empty list is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_model_replica_info
(name, version, replica_id)[source]¶ Gets detailed information about a registered container.
Parameters: Returns: A dictionary with the specified container’s info. If no corresponding container is registered with Clipper, None is returned.
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
get_clipper_logs
(logging_dir='clipper_logs/')[source]¶ Download the logs from all Clipper docker containers.
Parameters: logging_dir (str, optional) – The directory to save the downloaded logs. If the directory does not exist, it will be created. Raises: clipper.UnconnectedException
-
inspect_instance
()[source]¶ Fetches performance metrics from the running Clipper cluster.
Returns: The JSON string containing the current set of metrics for this instance. On error, the string will be an error message (not JSON formatted).
Return type: Raises: clipper.UnconnectedException
clipper.ClipperException
-
set_model_version
(name, version, num_replicas=None)[source]¶ Changes the current model version to “model_version”.
This method can be used to perform model roll-back and roll-forward. The version can be set to any previously deployed version of the model.
Parameters: Raises: clipper.UnconnectedException
clipper.ClipperException
Note
Model versions automatically get updated when py:meth:clipper_admin.ClipperConnection.deploy_model() is called. There is no need to manually update the version after deploying a new model.
-
get_query_addr
()[source]¶ Get the IP address at which the query frontend can be reached request predictions.
Returns: The address as an IP address or hostname. Return type: str Raises: clipper.UnconnectedException
– versions. All replicas for each version of each model will be stopped.
-
get_metric_addr
()[source]¶ Get the IP address of Prometheus metric server. :returns: The address as an IP address or hostname. :rtype: str
Raises: clipper.UnconnectedException
– versions. All replicas for each version of each model will be stopped.
-
stop_models
(model_names)[source]¶ Stops all versions of the specified models.
This is a convenience method to avoid the need to explicitly list all versions of a model when calling
clipper_admin.ClipperConnection.stop_versioned_models()
.Parameters: model_names (list(str)) – A list of model names. All replicas of all versions of each model specified in the list will be stopped. Raises: clipper.UnconnectedException
– versions. All replicas for each version of each model will be stopped.
-
stop_versioned_models
(model_versions_dict)[source]¶ Stops the specified versions of the specified models.
Parameters: model_versions_dict (dict(str, list(str))) – For each entry in the dict, the key is a model name and the value is a list of model Raises: clipper.UnconnectedException
– versions. All replicas for each version of each model will be stopped.Note
This method will stop the currently deployed versions of models if you specify them. You almost certainly want to use one of the other stop_* methods. Use with caution.
-
stop_inactive_model_versions
(model_names)[source]¶ Stops all model containers serving stale versions of the specified models.
For example, if you have deployed versions 1, 2, and 3 of model “music_recommender” and version 3 is the current version:
clipper_conn.stop_inactive_model_versions(["music_recommender"])
will stop any containers serving versions 1 and 2 but will leave containers serving version 3 untouched.
Parameters: model_names (list(str)) – The names of the models whose old containers you want to stop. Raises: clipper.UnconnectedException
-
stop_all_model_containers
()[source]¶ Stops all model containers started via Clipper admin commands.
This method can be used to clean up leftover Clipper model containers even if the Clipper management frontend or Redis has crashed. It can also be called without calling
connect
first.Raises: clipper.UnconnectedException
– versions. All replicas for each version of each model will be stopped.
-
stop_all
(graceful=True)[source]¶ Stops all processes that were started via Clipper admin commands.
This includes the query and management frontend Docker containers and all model containers. If you started Redis independently, this will not affect Redis. It can also be called without calling
connect
first.If graceful=False, Clipper will issue Docker Kill if it’s in the Docker Mode. This parameter will take not effect in Kubernetes.
-
test_predict_function
(query, func, input_type)[source]¶ Tests that the user’s function has the correct signature and can be properly saved and loaded.
The function should take a dict request object like the query frontend expects JSON, the predict function, and the input type for the model.
- For example, the function can be called like:
- clipper_conn.test_predict_function({“input”: [1.0, 2.0, 3.0]}, predict_func, “doubles”)
Parameters: - query (JSON or list of dicts) – Inputs to test the prediction function on.
- func (function) – Predict function to test.
- input_type (str) – The input_type to be associated with the registered app and deployed model. One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
-