Renderers

Collection of renderers

Example:

field_outputs = field(ray_sampler)
weights = ray_sampler.get_weights(field_outputs[FieldHeadNames.DENSITY])

rgb_renderer = RGBRenderer()
rgb = rgb_renderer(rgb=field_outputs[FieldHeadNames.RGB], weights=weights)

class nerfstudio.model_components.renderers.AccumulationRenderer(*args, **kwargs)

Bases: Module

Accumulated value along a ray.

classmethod forward(weights: Float[Tensor, '*bs num_samples 1'], ray_indices: Optional[Int[Tensor, 'num_samples']] = None, num_rays: Optional[int] = None) → Float[Tensor, '*bs 1']

Composite samples along ray and calculate accumulation.

Parameters:

• weights – Weights for each sample

• ray_indices – Ray index for each sample, used when samples are packed.

• num_rays – Number of rays, used when samples are packed.

Returns:

Outputs of accumulated values.

class nerfstudio.model_components.renderers.DepthRenderer(method: Literal['median', 'expected'] = 'median')

Bases: Module

Calculate depth along ray.

Depth Method:

• median: Depth is set to the distance where the accumulated weight reaches 0.5.

• expected: Expected depth along ray. Same procedure as rendering rgb, but with depth.

Parameters:

method – Depth calculation method.

forward(weights: Float[Tensor, '*batch num_samples 1'], ray_samples: RaySamples, ray_indices: Optional[Int[Tensor, 'num_samples']] = None, num_rays: Optional[int] = None) → Float[Tensor, '*batch 1']

Composite samples along ray and calculate depths.

Parameters:

• weights – Weights for each sample.

• ray_samples – Set of ray samples.

• ray_indices – Ray index for each sample, used when samples are packed.

• num_rays – Number of rays, used when samples are packed.

Returns:

Outputs of depth values.

class nerfstudio.model_components.renderers.NormalsRenderer(*args, **kwargs)

Bases: Module

Calculate normals along the ray.

classmethod forward(normals: Float[Tensor, '*bs num_samples 3'], weights: Float[Tensor, '*bs num_samples 1'], normalize: bool = True) → Float[Tensor, '*bs 3']

Calculate normals along the ray.

Parameters:

• normals – Normals for each sample.

• weights – Weights of each sample.

• normalize – Normalize normals.

class nerfstudio.model_components.renderers.RGBRenderer(background_color: Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']] = 'random')

Bases: Module

Standard volumetric rendering.

Parameters:

background_color – Background color as RGB. Uses random colors if None.

blend_background(image: Tensor, background_color: Optional[Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']]] = None) → Float[Tensor, '*bs 3']

Blends the background color into the image if image is RGBA. Otherwise no blending is performed (we assume opacity of 1).

Parameters:

• image – RGB/RGBA per pixel.

• opacity – Alpha opacity per pixel.

• background_color – Background color.

Returns:

Blended RGB.

blend_background_for_loss_computation(pred_image: Tensor, pred_accumulation: Tensor, gt_image: Tensor) → Tuple[Tensor, Tensor]

Blends a background color into the ground truth and predicted image for loss computation.

Parameters:

• gt_image – The ground truth image.

• pred_image – The predicted RGB values (without background blending).

• pred_accumulation – The predicted opacity/ accumulation.

Returns:

A tuple of the predicted and ground truth RGB values.

classmethod combine_rgb(rgb: Float[Tensor, '*bs num_samples 3'], weights: Float[Tensor, '*bs num_samples 1'], background_color: Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']] = 'random', ray_indices: Optional[Int[Tensor, 'num_samples']] = None, num_rays: Optional[int] = None) → Float[Tensor, '*bs 3']

Composite samples along ray and render color image. If background color is random, no BG color is added - as if the background was black!

Parameters:

• rgb – RGB for each sample

• weights – Weights for each sample

• background_color – Background color as RGB.

• ray_indices – Ray index for each sample, used when samples are packed.

• num_rays – Number of rays, used when samples are packed.

Returns:

Outputs rgb values.

forward(rgb: Float[Tensor, '*bs num_samples 3'], weights: Float[Tensor, '*bs num_samples 1'], ray_indices: Optional[Int[Tensor, 'num_samples']] = None, num_rays: Optional[int] = None, background_color: Optional[Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']]] = None) → Float[Tensor, '*bs 3']

Composite samples along ray and render color image

Parameters:

• rgb – RGB for each sample

• weights – Weights for each sample

• ray_indices – Ray index for each sample, used when samples are packed.

• num_rays – Number of rays, used when samples are packed.

• background_color – The background color to use for rendering.

Returns:

Outputs of rgb values.

classmethod get_background_color(background_color: Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']], shape: Tuple[int, ...], device: device) → Union[Float[Tensor, '3'], Float[Tensor, '*bs 3']]

Returns the RGB background color for a specified background color. .. note:: This function CANNOT be called for background_color being either “last_sample” or “random”.

Parameters:

• background_color – The background color specification. If a string is provided, it must be a valid color name.

• shape – Shape of the output tensor.

• device – Device on which to create the tensor.

Returns:

Background color as RGB.

class nerfstudio.model_components.renderers.SHRenderer(background_color: Union[Literal['random', 'last_sample', 'black', 'white'], Float[Tensor, '3'], Float[Tensor, '*bs 3']] = 'random', activation: Optional[Module] = Sigmoid())

Bases: Module

Render RGB value from spherical harmonics.

Parameters:

• background_color – Background color as RGB. Uses random colors if None

• activation – Output activation.

forward(sh: Float[Tensor, '*batch num_samples coeffs'], directions: Float[Tensor, '*batch num_samples 3'], weights: Float[Tensor, '*batch num_samples 1']) → Float[Tensor, '*batch 3']

Composite samples along ray and render color image

Parameters:

• sh – Spherical harmonics coefficients for each sample

• directions – Sample direction

• weights – Weights for each sample

Returns:

Outputs of rgb values.

class nerfstudio.model_components.renderers.SemanticRenderer(*args, **kwargs)

Bases: Module

Calculate semantics along the ray.

classmethod forward(semantics: Float[Tensor, '*bs num_samples num_classes'], weights: Float[Tensor, '*bs num_samples 1'], ray_indices: Optional[Int[Tensor, 'num_samples']] = None, num_rays: Optional[int] = None) → Float[Tensor, '*bs num_classes']

Calculate semantics along the ray.

class nerfstudio.model_components.renderers.UncertaintyRenderer(*args, **kwargs)

Bases: Module

Calculate uncertainty along the ray.

classmethod forward(betas: Float[Tensor, '*bs num_samples 1'], weights: Float[Tensor, '*bs num_samples 1']) → Float[Tensor, '*bs 1']

Calculate uncertainty along the ray.

Parameters:

• betas – Uncertainty betas for each sample.

• weights – Weights of each sample.

Returns:

Rendering of uncertainty.

nerfstudio.model_components.renderers.background_color_override_context(mode: Float[Tensor, '3']) → Generator[None, None, None]

Context manager for setting background mode.