Losses

Collection of Losses.

class nerfstudio.model_components.losses.DepthLossType(value)

Bases: Enum

Types of depth losses for depth supervision.

class nerfstudio.model_components.losses.GradientLoss(scales: int = 4, reduction_type: Literal['image', 'batch'] = 'batch')

Bases: Module

multiscale, scale-invariant gradient matching term to the disparity space. This term biases discontinuities to be sharp and to coincide with discontinuities in the ground truth More info here https://arxiv.org/pdf/1907.01341.pdf Equation 11

forward(prediction: Float[Tensor, '1 32 mult'], target: Float[Tensor, '1 32 mult'], mask: Bool[Tensor, '1 32 mult']) → Float[Tensor, '0']

Parameters:

• prediction – predicted depth map

• target – ground truth depth map

• mask – mask of valid pixels

Returns:

gradient loss based on reduction function

gradient_loss(prediction: Float[Tensor, '1 32 mult'], target: Float[Tensor, '1 32 mult'], mask: Bool[Tensor, '1 32 mult']) → Float[Tensor, '0']

multiscale, scale-invariant gradient matching term to the disparity space. This term biases discontinuities to be sharp and to coincide with discontinuities in the ground truth More info here https://arxiv.org/pdf/1907.01341.pdf Equation 11 :param prediction: predicted depth map :param target: ground truth depth map :param reduction: reduction function, either reduction_batch_based or reduction_image_based

Returns:

gradient loss based on reduction function

class nerfstudio.model_components.losses.MiDaSMSELoss(reduction_type: Literal['image', 'batch'] = 'batch')

Bases: Module

data term from MiDaS paper

forward(prediction: Float[Tensor, '1 32 mult'], target: Float[Tensor, '1 32 mult'], mask: Bool[Tensor, '1 32 mult']) → Float[Tensor, '0']

Parameters:

• prediction – predicted depth map

• target – ground truth depth map

• mask – mask of valid pixels

Returns:

mse loss based on reduction function

class nerfstudio.model_components.losses.ScaleAndShiftInvariantLoss(alpha: float = 0.5, scales: int = 4, reduction_type: Literal['image', 'batch'] = 'batch')

Bases: Module

Scale and shift invariant loss as described in “Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer” https://arxiv.org/pdf/1907.01341.pdf

forward(prediction: Float[Tensor, '1 32 mult'], target: Float[Tensor, '1 32 mult'], mask: Bool[Tensor, '1 32 mult']) → Float[Tensor, '0']

Parameters:

• prediction – predicted depth map (unnormalized)

• target – ground truth depth map (normalized)

• mask – mask of valid pixels

Returns:

scale and shift invariant loss

property prediction_ssi

scale and shift invariant prediction from https://arxiv.org/pdf/1907.01341.pdf equation 1

nerfstudio.model_components.losses.depth_loss(weights: Float[Tensor, '*batch num_samples 1'], ray_samples: RaySamples, termination_depth: Float[Tensor, '*batch 1'], predicted_depth: Float[Tensor, '*batch 1'], sigma: Float[Tensor, '0'], directions_norm: Float[Tensor, '*batch 1'], is_euclidean: bool, depth_loss_type: DepthLossType) → Float[Tensor, '0']

Implementation of depth losses.

Parameters:

• weights – Weights predicted for each sample.

• ray_samples – Samples along rays corresponding to weights.

• termination_depth – Ground truth depth of rays.

• predicted_depth – Depth prediction from the network.

• sigma – Uncertainty around depth value.

• directions_norm – Norms of ray direction vectors in the camera frame.

• is_euclidean – Whether ground truth depths corresponds to normalized direction vectors.

• depth_loss_type – Type of depth loss to apply.

Returns:

Depth loss scalar.

nerfstudio.model_components.losses.depth_ranking_loss(rendered_depth, gt_depth)

Depth ranking loss as described in the SparseNeRF paper Assumes that the layout of the batch comes from a PairPixelSampler, so that adjacent samples in the gt_depth and rendered_depth are from pixels with a radius of each other

nerfstudio.model_components.losses.distortion_loss(weights_list, ray_samples_list)

From mipnerf360

nerfstudio.model_components.losses.ds_nerf_depth_loss(weights: Float[Tensor, '*batch num_samples 1'], termination_depth: Float[Tensor, '*batch 1'], steps: Float[Tensor, '*batch num_samples 1'], lengths: Float[Tensor, '*batch num_samples 1'], sigma: Float[Tensor, '0']) → Float[Tensor, '*batch 1']

Depth loss from Depth-supervised NeRF (Deng et al., 2022).

Parameters:

• weights – Weights predicted for each sample.

• termination_depth – Ground truth depth of rays.

• steps – Sampling distances along rays.

• lengths – Distances between steps.

• sigma – Uncertainty around depth values.

Returns:

Depth loss scalar.

nerfstudio.model_components.losses.interlevel_loss(weights_list, ray_samples_list) → Tensor

Calculates the proposal loss in the MipNeRF-360 paper.

https://github.com/kakaobrain/NeRF-Factory/blob/f61bb8744a5cb4820a4d968fb3bfbed777550f4a/src/model/mipnerf360/model.py#L515 https://github.com/google-research/multinerf/blob/b02228160d3179300c7d499dca28cb9ca3677f32/internal/train_utils.py#L133

nerfstudio.model_components.losses.lossfun_distortion(t, w)

https://github.com/kakaobrain/NeRF-Factory/blob/f61bb8744a5cb4820a4d968fb3bfbed777550f4a/src/model/mipnerf360/helper.py#L142 https://github.com/google-research/multinerf/blob/b02228160d3179300c7d499dca28cb9ca3677f32/internal/stepfun.py#L266

nerfstudio.model_components.losses.lossfun_outer(t: Float[Tensor, '*batch num_samples_1'], w: Float[Tensor, '*batch num_samples'], t_env: Float[Tensor, '*batch num_samples_1'], w_env: Float[Tensor, '*batch num_samples'])

https://github.com/kakaobrain/NeRF-Factory/blob/f61bb8744a5cb4820a4d968fb3bfbed777550f4a/src/model/mipnerf360/helper.py#L136 https://github.com/google-research/multinerf/blob/b02228160d3179300c7d499dca28cb9ca3677f32/internal/stepfun.py#L80

Parameters:

• t – interval edges

• w – weights

• t_env – interval edges of the upper bound enveloping histogram

• w_env – weights that should upper bound the inner (t,w) histogram

nerfstudio.model_components.losses.monosdf_normal_loss(normal_pred: Float[Tensor, 'num_samples 3'], normal_gt: Float[Tensor, 'num_samples 3']) → Float[Tensor, '0']

Normal consistency loss proposed in monosdf - https://niujinshuchong.github.io/monosdf/ Enforces consistency between the volume rendered normal and the predicted monocular normal. With both angluar and L1 loss. Eq 14 https://arxiv.org/pdf/2206.00665.pdf :param normal_pred: volume rendered normal :param normal_gt: monocular normal

nerfstudio.model_components.losses.nerfstudio_distortion_loss(ray_samples: RaySamples, densities: Optional[Float[Tensor, '*bs num_samples 1']] = None, weights: Optional[Float[Tensor, '*bs num_samples 1']] = None) → Float[Tensor, '*bs 1']

Ray based distortion loss proposed in MipNeRF-360. Returns distortion Loss.

\[\mathcal{L}(\mathbf{s}, \mathbf{w}) =\iint\limits_{-\infty}^{\,\,\,\infty} \mathbf{w}_\mathbf{s}(u)\mathbf{w}_\mathbf{s}(v)|u - v|\,d_{u}\,d_{v}\]

where \(\mathbf{w}_\mathbf{s}(u)=\sum_i w_i \mathbb{1}_{[\mathbf{s}_i, \mathbf{s}_{i+1})}(u)\) is the weight at location \(u\) between bin locations \(s_i\) and \(s_{i+1}\).

Parameters:

• ray_samples – Ray samples to compute loss over

• densities – Predicted sample densities

• weights – Predicted weights from densities and sample locations

nerfstudio.model_components.losses.orientation_loss(weights: Float[Tensor, '*bs num_samples 1'], normals: Float[Tensor, '*bs num_samples 3'], viewdirs: Float[Tensor, '*bs 3'])

Orientation loss proposed in Ref-NeRF. Loss that encourages that all visible normals are facing towards the camera.

nerfstudio.model_components.losses.outer(t0_starts: Float[Tensor, '*batch num_samples_0'], t0_ends: Float[Tensor, '*batch num_samples_0'], t1_starts: Float[Tensor, '*batch num_samples_1'], t1_ends: Float[Tensor, '*batch num_samples_1'], y1: Float[Tensor, '*batch num_samples_1']) → Float[Tensor, '*batch num_samples_0']

Faster version of

https://github.com/kakaobrain/NeRF-Factory/blob/f61bb8744a5cb4820a4d968fb3bfbed777550f4a/src/model/mipnerf360/helper.py#L117 https://github.com/google-research/multinerf/blob/b02228160d3179300c7d499dca28cb9ca3677f32/internal/stepfun.py#L64

Parameters:

• t0_starts – start of the interval edges

• t0_ends – end of the interval edges

• t1_starts – start of the interval edges

• t1_ends – end of the interval edges

• y1 – weights

nerfstudio.model_components.losses.pred_normal_loss(weights: Float[Tensor, '*bs num_samples 1'], normals: Float[Tensor, '*bs num_samples 3'], pred_normals: Float[Tensor, '*bs num_samples 3'])

Loss between normals calculated from density and normals from prediction network.

nerfstudio.model_components.losses.ray_samples_to_sdist(ray_samples)

Convert ray samples to s space

nerfstudio.model_components.losses.scale_gradients_by_distance_squared(field_outputs: Dict[FieldHeadNames, Tensor], ray_samples: RaySamples) → Dict[FieldHeadNames, Tensor]

Scale gradients by the ray distance to the pixel as suggested in Radiance Field Gradient Scaling for Unbiased Near-Camera Training paper

Note: The scaling is applied on the interval of [0, 1] along the ray!

Example

GradientLoss should be called right after obtaining the densities and colors from the field. ::

>>> field_outputs = scale_gradient_by_distance_squared(field_outputs, ray_samples)

nerfstudio.model_components.losses.tv_loss(grids: Float[Tensor, 'grids feature_dim row column']) → Float[Tensor, '']

https://github.com/apchenstu/TensoRF/blob/4ec894dc1341a2201fe13ae428631b58458f105d/utils.py#L139

Parameters:

grids – stacks of explicit feature grids (stacked at dim 0)

Returns:

average total variation loss for neighbor rows and columns.

nerfstudio.model_components.losses.urban_radiance_field_depth_loss(weights: Float[Tensor, '*batch num_samples 1'], termination_depth: Float[Tensor, '*batch 1'], predicted_depth: Float[Tensor, '*batch 1'], steps: Float[Tensor, '*batch num_samples 1'], sigma: Float[Tensor, '0']) → Float[Tensor, '*batch 1']

Lidar losses from Urban Radiance Fields (Rematas et al., 2022).

Parameters:

• weights – Weights predicted for each sample.

• termination_depth – Ground truth depth of rays.

• predicted_depth – Depth prediction from the network.

• steps – Sampling distances along rays.

• sigma – Uncertainty around depth values.

Returns:

Depth loss scalar.