cerebras.pytorch.metrics#
Metrics#
A collection of evaluation metrics that can be used to evaluate the performance of a trained model on the Cerebras Wafer Scale Cluster.
Available metrics include:
metrics.Metric#
- class cerebras.pytorch.metrics.Metric(*args, **kwargs)[source]#
Base class for implementing metrics compatible with Cerebras WSC.
This class is designed to be used as a base class for implementing metrics compatible with Cerebras Wafer-Scale Cluster, but they also work with CPU and GPU backends.
To implement a new metric, subclass Metric and implement the following: - reset: This is to initialize the metric state. - update: This is to update the metric state at every iteration. - compute: This is to compute the final metric value based on the state.
To use metrics, instantiate them and call them with the appropriate inputs. For example:
>>> metric = MyMetric() >>> metric(input_1, input_2) # Calls update and compute >>> metric.compute() # Returns the final (cached) metric value
Constructs a Metric instance.
- Parameters
name (str) – The name of the metric. This is used to reference the metric and does not have to be unique.
- registry = {}#
- property num_updates: int#
Returns the number of times the metric was updated.
- register_state(name, tensor, persistent=False)[source]#
Registers a state variable to the module.
By default, metric state variables are non-persistent buffers that are not included in the module’s state dictionary. To have them as part of the state dictionary, set persistent=True.
Once registered, the state variable can be accessed as an attribute on the module by the given name.
- Parameters
name (str) – The name of the state variable.
tensor (torch.Tensor) – The tensor to register.
persistent (bool) – Whether this state is part of the module’s state_dict.
metrics.AccuracyMetric#
metrics.PerplexityMetric#
- class cerebras.pytorch.metrics.PerplexityMetric(*args, **kwargs)[source]#
Computes the perplexity of the model’s predictions
- Parameters
name – Name of the metric
Constructs a Metric instance.
- Parameters
name – The name of the metric. This is used to reference the metric and does not have to be unique.
metrics.DiceCoefficientMetric#
- class cerebras.pytorch.metrics.DiceCoefficientMetric(*args, **kwargs)[source]#
Dice Coefficient is a common evaluation metric for semantic image segmentation.
Dice Coefficient is defined as follows: Dice = 2 * true_positive / (2 * true_positive + false_positive + false_negative).
The predictions are accumulated in a confusion matrix, weighted by weights, and dice coefficient is then calculated from it.
- Parameters
num_classes – The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated.
name (Optional[str]) – Optional string which indicates name of the metric. If None or empty string, it defaults to the name of the class.
- update(labels, predictions, weights=None, dtype=None)[source]#
Updates the dice coefficient metric.
- Parameters
labels – A Tensor of ground truth labels of type int32 or int64.
predictions – A Tensor of prediction results for semantic labels, of type int32 or int64.
weights – Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding labels dimension). If weights is None, weights default to 1. Use weights of 0 to mask values.
- Raises
ValueError – If predictions and labels have mismatched shapes, or if weights is not None and its shape doesn’t match predictions
metrics.MeanIOUMetric#
- class cerebras.pytorch.metrics.MeanIOUMetric(*args, **kwargs)[source]#
Mean Intersection-Over-Union is a common evaluation metric for semantic image segmentation, which first computes the IOU for each semantic class and then computes the average over classes. iou is defined as follows: IOU = true_positive / (true_positive + false_positive + false_negative). The predictions are accumulated in a confusion matrix, weighted by weights, and mIOU is then calculated from it.
For estimation of the metric over a stream of data, the function creates an update_op operation that updates these variables and returns the mean_iou.
If weights is None, weights default to 1. Use weights of 0 to mask values.
- Parameters
num_classes – The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated.
name (Optional[str]) – Optional string which indicates name of the metric. If None or empty string, it defaults to the name of the class.
- update(labels, predictions, weights=None, dtype=None)[source]#
Updates the mean IOU metric.
- Parameters
labels – A Tensor of ground truth labels of type int32 or int64.
predictions – A Tensor of prediction results for semantic labels, of type int32 or int64.
weights – Optional Tensor whose rank is either 0, or the same rank as labels, and must be broadcastable to labels (i.e., all dimensions must be either 1, or the same as the corresponding labels dimension).
- Raises
ValueError – If predictions and labels have mismatched shapes, or if weights is not None and its shape doesn’t match predictions
metrics.FBetaScoreMetric#
- class cerebras.pytorch.metrics.FBetaScoreMetric(*args, **kwargs)[source]#
Calculates F Score from labels and predictions.
fbeta = (1 + beta^2) * (precision*recall) / ((beta^2 * precision) + recall)
Where beta is some positive real factor. :param num_classes: Number of classes. :param beta: Beta coefficient in the F measure. :param average_type: Defines the reduction that is applied. Should be one
of the following: - ‘micro’ [default]: Calculate the metric globally, across all
samples and classes.
- ‘macro’: Calculate the metric for each class separately, and
average the metrics across classes (with equal weights for each class). This does not take label imbalance into account.
- Parameters
ignore_labels (Optional[List]) – Integer specifying a target classes to ignore.
name (Optional[str]) – Name of the metric