Skip to content

metrics

Metrics for shape evaluation.

accuracy_thresh(points_gt, points_rec, threshold, p_norm=2, normalize=False)

Compute thresholded accuracy metric.

See FroDO: From Detections to 3D Objects, Rünz et al., 2020.

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
threshold float

distance threshold to count a point as correct

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide distances by Euclidean extent of points_gt

 False

Returns: Ratio of reconstructed points with closest ground truth point closer than threshold (in p-norm).

Source code in sdfest/estimation/metrics.py 
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
def accuracy_thresh(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    threshold: float,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute thresholded accuracy metric.

    See FroDO: From Detections to 3D Objects, Rünz et al., 2020.

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        threshold: distance threshold to count a point as correct
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide distances by Euclidean extent of points_gt
    Returns:
        Ratio of reconstructed points with closest ground truth point closer than
        threshold (in p-norm).
    """
    kd_tree = scipy.spatial.KDTree(points_gt)
    d, _ = kd_tree.query(points_rec, p=p_norm)
    if normalize:
        return np.sum(d / extent(points_gt) < threshold) / points_rec.shape[0]
    else:
        return np.sum(d < threshold) / points_rec.shape[0]

completeness_thresh(points_gt, points_rec, threshold, p_norm=2, normalize=False)

Compute thresholded completion metric.

See FroDO: From Detections to 3D Objects, Rünz et al., 2020.

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
threshold float

distance threshold to count a point as correct

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide distances by Euclidean extent of points_gt

 False

Returns: Ratio of ground truth points with closest reconstructed point closer than threshold (in p-norm).

Source code in sdfest/estimation/metrics.py 
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
def completeness_thresh(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    threshold: float,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute thresholded completion metric.

    See FroDO: From Detections to 3D Objects, Rünz et al., 2020.

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        threshold: distance threshold to count a point as correct
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide distances by Euclidean extent of points_gt
    Returns:
        Ratio of ground truth points with closest reconstructed point closer than
        threshold (in p-norm).
    """
    kd_tree = scipy.spatial.KDTree(points_rec)
    d, _ = kd_tree.query(points_gt, p=p_norm)
    if normalize:
        return np.sum(d / extent(points_gt) < threshold) / points_gt.shape[0]
    else:
        return np.sum(d < threshold) / points_gt.shape[0]

correct_thresh(position_gt, position_prediction, orientation_gt, orientation_prediction, extent_gt=None, extent_prediction=None, points_gt=None, points_prediction=None, position_threshold=None, degree_threshold=None, iou_3d_threshold=None, fscore_threshold=None, rotational_symmetry_axis=None)

Classify a pose prediction as correct or incorrect.

Parameters:

Name Type Description Default
position_gt ndarray

ground truth position, expected shape (3,).

 required
position_prediction ndarray

predicted position, expected shape (3,).

 required
position_threshold Optional[float]

position threshold in meters, no threshold if None

 None
orientation_q_qt

ground truth orientation, scalar-last quaternion, shape (4,)

 required
orientation_q_prediction

predicted orientation, scalar-last quaternion, shape (4,)

 required
extent_gt Optional[ndarray]

bounding box extents, shape (3,) only used if IoU threshold specified

 None
extent_prediction Optional[ndarray]

bounding box extents, shape (3,) only used if IoU threshold specified

 None
point_gt

set of true points, expected shape (N,3)

 required
points_rec

set of reconstructed points, expected shape (M,3)

 required
degree_threshold Optional[float]

orientation threshold in degrees, no threshold if None

 None
iou_3d_threshold Optional[float]

3D IoU threshold, no threshold if None

 None
rotational_symmetry_axis Optional[int]

Specify axis along which rotation is ignored. If None, no axis is ignored. 0 for x-axis, 1 for y-axis, 2 for z-axis.

 None

Returns: 1 if error is below all provided thresholds. 0 if error is above one provided threshold.

Source code in sdfest/estimation/metrics.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def correct_thresh(
    position_gt: np.ndarray,
    position_prediction: np.ndarray,
    orientation_gt: Rotation,
    orientation_prediction: Rotation,
    extent_gt: Optional[np.ndarray] = None,
    extent_prediction: Optional[np.ndarray] = None,
    points_gt: Optional[np.ndarray] = None,
    points_prediction: Optional[np.ndarray] = None,
    position_threshold: Optional[float] = None,
    degree_threshold: Optional[float] = None,
    iou_3d_threshold: Optional[float] = None,
    fscore_threshold: Optional[float] = None,
    rotational_symmetry_axis: Optional[int] = None,
) -> int:
    """Classify a pose prediction as correct or incorrect.

    Args:
        position_gt: ground truth position, expected shape (3,).
        position_prediction: predicted position, expected shape (3,).
        position_threshold: position threshold in meters, no threshold if None
        orientation_q_qt: ground truth orientation, scalar-last quaternion, shape (4,)
        orientation_q_prediction:
            predicted orientation, scalar-last quaternion, shape (4,)
        extent_gt:
            bounding box extents, shape (3,)
            only used if IoU threshold specified
        extent_prediction:
            bounding box extents, shape (3,)
            only used if IoU threshold specified
        point_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        degree_threshold: orientation threshold in degrees, no threshold if None
        iou_3d_threshold: 3D IoU threshold, no threshold if None
        rotational_symmetry_axis:
            Specify axis along which rotation is ignored. If None, no axis is ignored.
            0 for x-axis, 1 for y-axis, 2 for z-axis.
    Returns:
        1 if error is below all provided thresholds.  0 if error is above one provided
        threshold.
    """
    if position_threshold is not None:
        position_error = np.linalg.norm(position_gt - position_prediction)
        if position_error > position_threshold:
            return 0
    if degree_threshold is not None:
        rad_threshold = degree_threshold * np.pi / 180.0
        if rotational_symmetry_axis is not None:
            p = np.array([0.0, 0.0, 0.0])
            p[rotational_symmetry_axis] = 1.0
            p1 = orientation_gt.apply(p)
            p2 = orientation_prediction.apply(p)
            rad_error = np.arccos(p1 @ p2)
        else:
            rad_error = (orientation_gt * orientation_prediction.inv()).magnitude()
        if rad_error > rad_threshold:
            return 0
    if iou_3d_threshold is not None:
        raise NotImplementedError("3D IoU is not impemented yet.")
        # TODO implement 3D IoU
        # starting point for proper implementation: https://github.com/google-research-datasets/Objectron/blob/c06a65165a18396e1e00091981fd1652875c97b5/objectron/dataset/iou.py#L6
        pass
    if fscore_threshold is not None:
        fscore = reconstruction_fscore(points_gt, points_prediction, 0.01)
        if fscore < fscore_threshold:
            return 0
    return 1

extent(points)

Compute largest Euclidean distance between any two points.

Parameters:

Name Type Description Default
points_gt

set of true

 required
p_norm

which Minkowski p-norm is used for distance and nearest neighbor query

 required

Returns: Ratio of reconstructed points with closest ground truth point closer than threshold (in p-norm).

Source code in sdfest/estimation/metrics.py 
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
def extent(points: np.ndarray) -> float:
    """Compute largest Euclidean distance between any two points.

    Args:
        points_gt: set of true
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
    Returns:
        Ratio of reconstructed points with closest ground truth point closer than
        threshold (in p-norm).
    """
    try:
        hull = scipy.spatial.ConvexHull(points)
    except scipy.spatial.qhull.QhullError:
        # fallback to brute force distance matrix
        return np.max(scipy.spatial.distance_matrix(points, points))

    # this is wasteful, if too slow implement rotating caliper method
    return np.max(
        scipy.spatial.distance_matrix(points[hull.vertices], points[hull.vertices])
    )

mean_accuracy(points_gt, points_rec, p_norm=2, normalize=False)

Compute accuracy metric.

Accuracy metric is the same as asymmetric chamfer distance from rec to gt.

See, for example, Occupancy Networks Learning 3D Reconstruction in Function Space, Mescheder et al., 2019.

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide result by Euclidean extent of points_gt

 False

Returns: Arithmetic mean of p-norm from reconstructed points to closest (in p-norm) ground truth points.

Source code in sdfest/estimation/metrics.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def mean_accuracy(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute accuracy metric.

    Accuracy metric is the same as asymmetric chamfer distance from rec to gt.

    See, for example, Occupancy Networks Learning 3D Reconstruction in Function Space,
    Mescheder et al., 2019.

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide result by Euclidean extent of points_gt
    Returns:
        Arithmetic mean of p-norm from reconstructed points to closest (in p-norm)
        ground truth points.
    """
    kd_tree = scipy.spatial.KDTree(points_gt)
    d, _ = kd_tree.query(points_rec, p=p_norm)
    if normalize:
        return np.mean(d) / extent(points_gt)
    else:
        return np.mean(d)

mean_completeness(points_gt, points_rec, p_norm=2, normalize=False)

Compute completeness metric.

Completeness metric is the same as asymmetric chamfer distance from gt to rec.

See, for example, Occupancy Networks Learning 3D Reconstruction in Function Space, Mescheder et al., 2019.

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide result by Euclidean extent of points_gt

 False

Returns: Arithmetic mean of p-norm from ground truth points to closest (in p-norm) reconstructed points.

Source code in sdfest/estimation/metrics.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
def mean_completeness(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute completeness metric.

    Completeness metric is the same as asymmetric chamfer distance from gt to rec.

    See, for example, Occupancy Networks Learning 3D Reconstruction in Function Space,
    Mescheder et al., 2019.

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide result by Euclidean extent of points_gt
    Returns:
        Arithmetic mean of p-norm from ground truth points to closest (in p-norm)
        reconstructed points.
    """
    kd_tree = scipy.spatial.KDTree(points_rec)
    d, _ = kd_tree.query(points_gt, p=p_norm)
    if normalize:
        return np.mean(d) / extent(points_gt)
    else:
        return np.mean(d)

reconstruction_fscore(points_gt, points_rec, threshold, p_norm=2, normalize=False)

Compute reconstruction fscore.

See What Do Single-View 3D Reconstruction Networks Learn, Tatarchenko, 2019

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
threshold float

distance threshold to count a point as correct

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide distances by Euclidean extent of points_gt

 False

Returns: Harmonic mean of precision (thresholded accuracy) and recall (thresholded completeness).

Source code in sdfest/estimation/metrics.py 
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
def reconstruction_fscore(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    threshold: float,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute reconstruction fscore.

    See What Do Single-View 3D Reconstruction Networks Learn, Tatarchenko, 2019

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        threshold: distance threshold to count a point as correct
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide distances by Euclidean extent of points_gt
    Returns:
        Harmonic mean of precision (thresholded accuracy) and recall (thresholded
        completeness).
    """
    recall = completeness_thresh(
        points_gt, points_rec, threshold, p_norm=p_norm, normalize=normalize
    )
    precision = accuracy_thresh(
        points_gt, points_rec, threshold, p_norm=p_norm, normalize=normalize
    )
    if recall < 1e-7 or precision < 1e-7:
        return 0
    return 2 / (1 / recall + 1 / precision)

symmetric_chamfer(points_gt, points_rec, p_norm=2, normalize=False)

Compute symmetric chamfer distance.

There are various slightly different definitions for the chamfer distance.

Note that completeness and accuracy are themselves sometimes referred to as chamfer distances, with symmetric chamfer distance being the combination of the two.

Chamfer L1 in the literature (see, for example, Occupancy Networks Learning 3D Reconstruction in Function Space, Mescheder et al., 2019) refers to using arithmetic mean (note that this is actually differently scaled from L1) when combining accuracy and completeness.

Parameters:

Name Type Description Default
points_gt ndarray

set of true points, expected shape (N,3)

 required
points_rec ndarray

set of reconstructed points, expected shape (M,3)

 required
p_norm int

which Minkowski p-norm is used for distance and nearest neighbor query

 2
normalize bool

whether to divide result by Euclidean extent of points_gt

 False

Returns: Arithmetic mean of accuracy and completeness metrics using the specified p-norm.

Source code in sdfest/estimation/metrics.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def symmetric_chamfer(
    points_gt: np.ndarray,
    points_rec: np.ndarray,
    p_norm: int = 2,
    normalize: bool = False,
) -> float:
    """Compute symmetric chamfer distance.

    There are various slightly different definitions for the chamfer distance.

    Note that completeness and accuracy are themselves sometimes referred to as
    chamfer distances, with symmetric chamfer distance being the combination of the two.

    Chamfer L1 in the literature (see, for example, Occupancy Networks Learning 3D
    Reconstruction in Function Space, Mescheder et al., 2019) refers to using
    arithmetic mean (note that this is actually differently scaled from L1) when
    combining accuracy and completeness.

    Args:
        points_gt: set of true points, expected shape (N,3)
        points_rec: set of reconstructed points, expected shape (M,3)
        p_norm: which Minkowski p-norm is used for distance and nearest neighbor query
        normalize: whether to divide result by Euclidean extent of points_gt
    Returns:
        Arithmetic mean of accuracy and completeness metrics using the specified p-norm.
    """
    return (
        mean_completeness(points_gt, points_rec, p_norm=p_norm, normalize=normalize)
        + mean_accuracy(points_gt, points_rec, p_norm=p_norm, normalize=normalize)
    ) / 2