Fields

Base

Base class for the graphs.

class nerfstudio.fields.base_field.Field

Bases: Module

Base class for fields.

density_fn(positions: Shaped[Tensor, '*bs 3'], times: Optional[Shaped[Tensor, '*bs 1']] = None) → Shaped[Tensor, '*bs 1']

Returns only the density. Used primarily with the density grid.

Parameters:

positions – the origin of the samples/frustums

forward(ray_samples: RaySamples, compute_normals: bool = False) → Dict[FieldHeadNames, Tensor]

Evaluates the field at points along the ray.

Parameters:

ray_samples – Samples to evaluate field on.

abstract get_density(ray_samples: RaySamples) → Tuple[Shaped[Tensor, '*batch 1'], Float[Tensor, '*batch num_features']]

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_normals() → Float[Tensor, '*batch 3']

Computes and returns a tensor of normals.

Parameters:

density – Tensor of densities.

abstract get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → Dict[FieldHeadNames, Tensor]

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.

class nerfstudio.fields.base_field.FieldConfig(_target: ~typing.Type = <factory>)

Bases: InstantiateConfig

Configuration for field instantiation

nerfstudio.fields.base_field.get_normalized_directions(directions: Float[Tensor, '*bs 3']) → Float[Tensor, '*bs 3']

SH encoding must be in the range [0, 1]

Parameters:

directions – batch of directions

Density

Proposal network field.

class nerfstudio.fields.density_fields.HashMLPDensityField(aabb: Tensor, num_layers: int = 2, hidden_dim: int = 64, spatial_distortion: Optional[SpatialDistortion] = None, use_linear: bool = False, num_levels: int = 8, max_res: int = 1024, base_res: int = 16, log2_hashmap_size: int = 18, features_per_level: int = 2, average_init_density: float = 1.0, implementation: Literal['tcnn', 'torch'] = 'tcnn')

Bases: Field

A lightweight density field module.

Parameters:

• aabb – parameters of scene aabb bounds

• num_layers – number of hidden layers

• hidden_dim – dimension of hidden layers

• spatial_distortion – spatial distortion module

• use_linear – whether to skip the MLP and use a single linear layer instead

get_density(ray_samples: RaySamples) → Tuple[Tensor, None]

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → dict

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.

Nerfacto

Field for compound nerf model, adds scene contraction and image embeddings to instant ngp

class nerfstudio.fields.nerfacto_field.NerfactoField(aabb: Tensor, num_images: int, num_layers: int = 2, hidden_dim: int = 64, geo_feat_dim: int = 15, num_levels: int = 16, base_res: int = 16, max_res: int = 2048, log2_hashmap_size: int = 19, num_layers_color: int = 3, num_layers_transient: int = 2, features_per_level: int = 2, hidden_dim_color: int = 64, hidden_dim_transient: int = 64, appearance_embedding_dim: int = 32, transient_embedding_dim: int = 16, use_transient_embedding: bool = False, use_semantics: bool = False, num_semantic_classes: int = 100, pass_semantic_gradients: bool = False, use_pred_normals: bool = False, use_average_appearance_embedding: bool = False, spatial_distortion: Optional[SpatialDistortion] = None, average_init_density: float = 1.0, implementation: Literal['tcnn', 'torch'] = 'tcnn')

Bases: Field

Compound Field

Parameters:

• aabb – parameters of scene aabb bounds

• num_images – number of images in the dataset

• num_layers – number of hidden layers

• hidden_dim – dimension of hidden layers

• geo_feat_dim – output geo feat dimensions

• num_levels – number of levels of the hashmap for the base mlp

• base_res – base resolution of the hashmap for the base mlp

• max_res – maximum resolution of the hashmap for the base mlp

• log2_hashmap_size – size of the hashmap for the base mlp

• num_layers_color – number of hidden layers for color network

• num_layers_transient – number of hidden layers for transient network

• features_per_level – number of features per level for the hashgrid

• hidden_dim_color – dimension of hidden layers for color network

• hidden_dim_transient – dimension of hidden layers for transient network

• appearance_embedding_dim – dimension of appearance embedding

• transient_embedding_dim – dimension of transient embedding

• use_transient_embedding – whether to use transient embedding

• use_semantics – whether to use semantic segmentation

• num_semantic_classes – number of semantic classes

• use_pred_normals – whether to use predicted normals

• use_average_appearance_embedding – whether to use average appearance embedding or zeros for inference

• spatial_distortion – spatial distortion to apply to the scene

get_density(ray_samples: RaySamples) → Tuple[Tensor, Tensor]

Computes and returns the densities.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → Dict[FieldHeadNames, Tensor]

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.

Nerf-W

Fields for nerf-w

class nerfstudio.fields.nerfw_field.VanillaNerfWField(num_images: int, position_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, appearance_embedding_dim: int = 48, transient_embedding_dim: int = 16, skip_connections: Tuple[int] = (4,))

Bases: Field

The NeRF-W field which has appearance and transient conditioning.

Parameters:

• num_images – How many images exist in the dataset.

• position_encoding – Position encoder.

• direction_encoding – Direction encoder.

• base_mlp_num_layers – Number of layers for base MLP.

• base_mlp_layer_width – Width of base MLP layers.

• head_mlp_num_layers – Number of layer for output head MLP.

• head_mlp_layer_width – Width of output head MLP layers.

• appearance_embedding_dim: – Dimension of appearance embedding.:

• transient_embedding_dim: – Dimension of transient embedding.:

• skip_connections – Where to add skip connection in base MLP.

get_density(ray_samples: RaySamples) → Tuple[Tensor, Tensor]

Computes and returns the densities.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Float[Tensor, '*batch embedding_size']] = None) → Dict[FieldHeadNames, Tensor]

Returns the outputs of the NeRF-W field.

Parameters:

• ray_samples – Ray samples.

• density_embedding – Density embedding.

Returns:

Outputs of the NeRF-W field.

SDF

Field for SDF based model, rather then estimating density to generate a surface, a signed distance function (SDF) for surface representation is used to help with extracting high fidelity surfaces

class nerfstudio.fields.sdf_field.LearnedVariance(init_val)

Bases: Module

Variance network in NeuS

Parameters:

init_val – initial value in NeuS variance network

forward(x: Float[Tensor, '1']) → Float[Tensor, '1']

Returns current variance value

get_variance() → Float[Tensor, '1']

return current variance value

class nerfstudio.fields.sdf_field.SDFField(config: SDFFieldConfig, aabb: Float[Tensor, '2 3'], num_images: int, use_average_appearance_embedding: bool = False, spatial_distortion: Optional[SpatialDistortion] = None)

Bases: Field

A field for Signed Distance Functions (SDF).

Parameters:

• config – The configuration for the SDF field.

• aabb – An axis-aligned bounding box for the SDF field.

• num_images – The number of images for embedding appearance.

• use_average_appearance_embedding – Whether to use average appearance embedding. Defaults to False.

• spatial_distortion – The spatial distortion. Defaults to None.

forward(ray_samples: RaySamples, compute_normals: bool = False, return_alphas: bool = False) → Dict[FieldHeadNames, Tensor]

Evaluates the field at points along the ray.

Parameters:

• ray_samples – Samples to evaluate field on.

• normals (compute) – not currently used in this implementation.

• return_alphas – Whether to return alpha values

forward_geonetwork(inputs: Float[Tensor, '*batch 3']) → Float[Tensor, '*batch geo_features+1']

forward the geonetwork

get_alpha(ray_samples: RaySamples, sdf: Optional[Float[Tensor, 'num_samples ... 1']] = None, gradients: Optional[Float[Tensor, 'num_samples ... 1']] = None) → Float[Tensor, 'num_samples ... 1']

compute alpha from sdf as in NeuS

get_colors(points: Float[Tensor, '*batch 3'], directions: Float[Tensor, '*batch 3'], normals: Float[Tensor, '*batch 3'], geo_features: Float[Tensor, '*batch geo_feat_dim'], camera_indices: Tensor) → Float[Tensor, '*batch 3']

compute colors

get_density(ray_samples: RaySamples)

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None, return_alphas: bool = False) → Dict[FieldHeadNames, Tensor]

compute output of ray samples

get_sdf(ray_samples: RaySamples) → Float[Tensor, 'num_samples ... 1']

predict the sdf value for ray samples

initialize_geo_layers() → None

Initialize layers for geometric network (sdf)

set_cos_anneal_ratio(anneal: float) → None

Set the anneal value for the proposal network.

class nerfstudio.fields.sdf_field.SDFFieldConfig(_target: ~typing.Type = <factory>, num_layers: int = 8, hidden_dim: int = 256, geo_feat_dim: int = 256, num_layers_color: int = 4, hidden_dim_color: int = 256, appearance_embedding_dim: int = 32, use_appearance_embedding: bool = False, bias: float = 0.8, geometric_init: bool = True, inside_outside: bool = True, weight_norm: bool = True, use_grid_feature: bool = False, divide_factor: float = 2.0, beta_init: float = 0.1, encoding_type: ~typing.Literal['hash', 'periodic', 'tensorf_vm'] = 'hash', num_levels: int = 16, max_res: int = 2048, base_res: int = 16, log2_hashmap_size: int = 19, features_per_level: int = 2, use_hash: bool = True, smoothstep: bool = True)

Bases: FieldConfig

SDF Field Config

appearance_embedding_dim: int = 32

Dimension of appearance embedding

base_res: int = 16

Base resolution of the encoding

beta_init: float = 0.1

Init learnable beta value for transformation of sdf to density

bias: float = 0.8

Sphere size of geometric initialization

divide_factor: float = 2.0

Normalization factor for multi-resolution grids

features_per_level: int = 2

Number of features per encoding level

geo_feat_dim: int = 256

Dimension of geometric feature

geometric_init: bool = True

Whether to use geometric initialization

hidden_dim: int = 256

Number of hidden dimension of geometric network

hidden_dim_color: int = 256

Number of hidden dimension of color network

inside_outside: bool = True

Whether to revert signed distance value, set to True for indoor scene

log2_hashmap_size: int = 19

Size of the hash map

max_res: int = 2048

Maximum resolution of the encoding

num_layers: int = 8

Number of layers for geometric network

num_layers_color: int = 4

Number of layers for color network

num_levels: int = 16

Number of encoding levels

smoothstep: bool = True

Whether to use the smoothstep function

use_appearance_embedding: bool = False

Whether to use appearance embedding

use_grid_feature: bool = False

Whether to use multi-resolution feature grids

use_hash: bool = True

Whether to use hash encoding

weight_norm: bool = True

Whether to use weight norm for linear layer

Semantic NeRF

Semantic NeRF field implementation.

class nerfstudio.fields.semantic_nerf_field.SemanticNerfField(num_semantic_classes: int, position_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, skip_connections: Tuple[int] = (4,))

Bases: Field

Semantic-NeRF field

Parameters:

• num_semantic_classes – Number of distinct semantic classes.

• position_encoding – Position encoder.

• direction_encoding – Direction encoder.

• base_mlp_num_layers – Number of layers for base MLP.

• base_mlp_layer_width – Width of base MLP layers.

• head_mlp_num_layers – Number of layer for output head MLP.

• head_mlp_layer_width – Width of output head MLP layers.

• skip_connections – Where to add skip connection in base MLP.

get_density(ray_samples: RaySamples) → Tuple[Tensor, Tensor]

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → Dict[FieldHeadNames, Tensor]

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.

TensoRF

TensoRF Field

class nerfstudio.fields.tensorf_field.TensoRFField(aabb: Tensor, feature_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), density_encoding: Encoding = Identity(), color_encoding: Encoding = Identity(), appearance_dim: int = 27, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, use_sh: bool = False, sh_levels: int = 2)

Bases: Field

TensoRF Field

forward(ray_samples: RaySamples, compute_normals: bool = False, mask: Optional[Tensor] = None, bg_color: Optional[Tensor] = None) → Dict[FieldHeadNames, Tensor]

Evaluates the field at points along the ray.

Parameters:

ray_samples – Samples to evaluate field on.

get_density(ray_samples: RaySamples) → Tensor

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → Tensor

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.

Vanilla NeRF

Classic NeRF field

class nerfstudio.fields.vanilla_nerf_field.NeRFField(position_encoding: ~nerfstudio.field_components.encodings.Encoding = Identity(), direction_encoding: ~nerfstudio.field_components.encodings.Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, skip_connections: ~typing.Tuple[int] = (4, ), field_heads: ~typing.Optional[~typing.Tuple[~typing.Type[~nerfstudio.field_components.field_heads.FieldHead]]] = (<class 'nerfstudio.field_components.field_heads.RGBFieldHead'>, ), use_integrated_encoding: bool = False, spatial_distortion: ~typing.Optional[~nerfstudio.field_components.spatial_distortions.SpatialDistortion] = None)

Bases: Field

NeRF Field

Parameters:

• position_encoding – Position encoder.

• direction_encoding – Direction encoder.

• base_mlp_num_layers – Number of layers for base MLP.

• base_mlp_layer_width – Width of base MLP layers.

• head_mlp_num_layers – Number of layer for output head MLP.

• head_mlp_layer_width – Width of output head MLP layers.

• skip_connections – Where to add skip connection in base MLP.

• use_integrated_encoding – Used integrated samples as encoding input.

• spatial_distortion – Spatial distortion.

get_density(ray_samples: RaySamples) → Tuple[Tensor, Tensor]

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) → Dict[FieldHeadNames, Tensor]

Computes and returns the colors. Returns output field values.

Parameters:

• ray_samples – Samples locations to compute outputs.

• density_embedding – Density embeddings to condition on.