Fields#

Base#

Base class for the graphs.

class nerfstudio.fields.base_field.Field[source]#

Bases: Module

Base class for fields.

density_fn(positions: Shaped[Tensor, '*bs 3'], times: Optional[Shaped[Tensor, '*bs 1']] = None) Shaped[Tensor, '*bs 1'][source]#

Returns only the density. Used primarily with the density grid.

Parameters:

positions – the origin of the samples/frustums

forward(ray_samples: RaySamples, compute_normals: bool = False) Dict[FieldHeadNames, Tensor][source]#

Evaluates the field at points along the ray.

Parameters:

ray_samples – Samples to evaluate field on.

abstract get_density(ray_samples: RaySamples) Tuple[Shaped[Tensor, '*batch 1'], Float[Tensor, '*batch num_features']][source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_normals() Float[Tensor, '*batch 3'][source]#

Computes and returns a tensor of normals.

Parameters:

density – Tensor of densities.

abstract get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) Dict[FieldHeadNames, Tensor][source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.

class nerfstudio.fields.base_field.FieldConfig(_target: ~typing.Type = <factory>)[source]#

Bases: InstantiateConfig

Configuration for field instantiation

nerfstudio.fields.base_field.get_normalized_directions(directions: Float[Tensor, '*bs 3']) Float[Tensor, '*bs 3'][source]#

SH encoding must be in the range [0, 1]

Parameters:

directions – batch of directions

Density#

Proposal network field.

class nerfstudio.fields.density_fields.HashMLPDensityField(aabb: Tensor, num_layers: int = 2, hidden_dim: int = 64, spatial_distortion: Optional[SpatialDistortion] = None, use_linear: bool = False, num_levels: int = 8, max_res: int = 1024, base_res: int = 16, log2_hashmap_size: int = 18, features_per_level: int = 2, average_init_density: float = 1.0, implementation: Literal['tcnn', 'torch'] = 'tcnn')[source]#

Bases: Field

A lightweight density field module.

Parameters:
  • aabb – parameters of scene aabb bounds

  • num_layers – number of hidden layers

  • hidden_dim – dimension of hidden layers

  • spatial_distortion – spatial distortion module

  • use_linear – whether to skip the MLP and use a single linear layer instead

get_density(ray_samples: RaySamples) Tuple[Tensor, None][source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) dict[source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.

Nerfacto#

Field for compound nerf model, adds scene contraction and image embeddings to instant ngp

class nerfstudio.fields.nerfacto_field.NerfactoField(aabb: Tensor, num_images: int, num_layers: int = 2, hidden_dim: int = 64, geo_feat_dim: int = 15, num_levels: int = 16, base_res: int = 16, max_res: int = 2048, log2_hashmap_size: int = 19, num_layers_color: int = 3, num_layers_transient: int = 2, features_per_level: int = 2, hidden_dim_color: int = 64, hidden_dim_transient: int = 64, appearance_embedding_dim: int = 32, transient_embedding_dim: int = 16, use_transient_embedding: bool = False, use_semantics: bool = False, num_semantic_classes: int = 100, pass_semantic_gradients: bool = False, use_pred_normals: bool = False, use_average_appearance_embedding: bool = False, spatial_distortion: Optional[SpatialDistortion] = None, average_init_density: float = 1.0, implementation: Literal['tcnn', 'torch'] = 'tcnn')[source]#

Bases: Field

Compound Field

Parameters:
  • aabb – parameters of scene aabb bounds

  • num_images – number of images in the dataset

  • num_layers – number of hidden layers

  • hidden_dim – dimension of hidden layers

  • geo_feat_dim – output geo feat dimensions

  • num_levels – number of levels of the hashmap for the base mlp

  • base_res – base resolution of the hashmap for the base mlp

  • max_res – maximum resolution of the hashmap for the base mlp

  • log2_hashmap_size – size of the hashmap for the base mlp

  • num_layers_color – number of hidden layers for color network

  • num_layers_transient – number of hidden layers for transient network

  • features_per_level – number of features per level for the hashgrid

  • hidden_dim_color – dimension of hidden layers for color network

  • hidden_dim_transient – dimension of hidden layers for transient network

  • appearance_embedding_dim – dimension of appearance embedding

  • transient_embedding_dim – dimension of transient embedding

  • use_transient_embedding – whether to use transient embedding

  • use_semantics – whether to use semantic segmentation

  • num_semantic_classes – number of semantic classes

  • use_pred_normals – whether to use predicted normals

  • use_average_appearance_embedding – whether to use average appearance embedding or zeros for inference

  • spatial_distortion – spatial distortion to apply to the scene

get_density(ray_samples: RaySamples) Tuple[Tensor, Tensor][source]#

Computes and returns the densities.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) Dict[FieldHeadNames, Tensor][source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.

Nerf-W#

Fields for nerf-w

class nerfstudio.fields.nerfw_field.VanillaNerfWField(num_images: int, position_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, appearance_embedding_dim: int = 48, transient_embedding_dim: int = 16, skip_connections: Tuple[int] = (4,))[source]#

Bases: Field

The NeRF-W field which has appearance and transient conditioning.

Parameters:
  • num_images – How many images exist in the dataset.

  • position_encoding – Position encoder.

  • direction_encoding – Direction encoder.

  • base_mlp_num_layers – Number of layers for base MLP.

  • base_mlp_layer_width – Width of base MLP layers.

  • head_mlp_num_layers – Number of layer for output head MLP.

  • head_mlp_layer_width – Width of output head MLP layers.

  • appearance_embedding_dim: – Dimension of appearance embedding.:

  • transient_embedding_dim: – Dimension of transient embedding.:

  • skip_connections – Where to add skip connection in base MLP.

get_density(ray_samples: RaySamples) Tuple[Tensor, Tensor][source]#

Computes and returns the densities.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Float[Tensor, '*batch embedding_size']] = None) Dict[FieldHeadNames, Tensor][source]#

Returns the outputs of the NeRF-W field.

Parameters:
  • ray_samples – Ray samples.

  • density_embedding – Density embedding.

Returns:

Outputs of the NeRF-W field.

SDF#

Field for SDF based model, rather then estimating density to generate a surface, a signed distance function (SDF) for surface representation is used to help with extracting high fidelity surfaces

class nerfstudio.fields.sdf_field.LearnedVariance(init_val)[source]#

Bases: Module

Variance network in NeuS

Parameters:

init_val – initial value in NeuS variance network

forward(x: Float[Tensor, '1']) Float[Tensor, '1'][source]#

Returns current variance value

get_variance() Float[Tensor, '1'][source]#

return current variance value

class nerfstudio.fields.sdf_field.SDFField(config: SDFFieldConfig, aabb: Float[Tensor, '2 3'], num_images: int, use_average_appearance_embedding: bool = False, spatial_distortion: Optional[SpatialDistortion] = None)[source]#

Bases: Field

A field for Signed Distance Functions (SDF).

Parameters:
  • config – The configuration for the SDF field.

  • aabb – An axis-aligned bounding box for the SDF field.

  • num_images – The number of images for embedding appearance.

  • use_average_appearance_embedding – Whether to use average appearance embedding. Defaults to False.

  • spatial_distortion – The spatial distortion. Defaults to None.

forward(ray_samples: RaySamples, compute_normals: bool = False, return_alphas: bool = False) Dict[FieldHeadNames, Tensor][source]#

Evaluates the field at points along the ray.

Parameters:
  • ray_samples – Samples to evaluate field on.

  • normals (compute) – not currently used in this implementation.

  • return_alphas – Whether to return alpha values

forward_geonetwork(inputs: Float[Tensor, '*batch 3']) Float[Tensor, '*batch geo_features+1'][source]#

forward the geonetwork

get_alpha(ray_samples: RaySamples, sdf: Optional[Float[Tensor, 'num_samples ... 1']] = None, gradients: Optional[Float[Tensor, 'num_samples ... 1']] = None) Float[Tensor, 'num_samples ... 1'][source]#

compute alpha from sdf as in NeuS

get_colors(points: Float[Tensor, '*batch 3'], directions: Float[Tensor, '*batch 3'], normals: Float[Tensor, '*batch 3'], geo_features: Float[Tensor, '*batch geo_feat_dim'], camera_indices: Tensor) Float[Tensor, '*batch 3'][source]#

compute colors

get_density(ray_samples: RaySamples)[source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None, return_alphas: bool = False) Dict[FieldHeadNames, Tensor][source]#

compute output of ray samples

get_sdf(ray_samples: RaySamples) Float[Tensor, 'num_samples ... 1'][source]#

predict the sdf value for ray samples

initialize_geo_layers() None[source]#

Initialize layers for geometric network (sdf)

set_cos_anneal_ratio(anneal: float) None[source]#

Set the anneal value for the proposal network.

class nerfstudio.fields.sdf_field.SDFFieldConfig(_target: ~typing.Type = <factory>, num_layers: int = 8, hidden_dim: int = 256, geo_feat_dim: int = 256, num_layers_color: int = 4, hidden_dim_color: int = 256, appearance_embedding_dim: int = 32, use_appearance_embedding: bool = False, bias: float = 0.8, geometric_init: bool = True, inside_outside: bool = True, weight_norm: bool = True, use_grid_feature: bool = False, divide_factor: float = 2.0, beta_init: float = 0.1, encoding_type: ~typing.Literal['hash', 'periodic', 'tensorf_vm'] = 'hash', num_levels: int = 16, max_res: int = 2048, base_res: int = 16, log2_hashmap_size: int = 19, features_per_level: int = 2, use_hash: bool = True, smoothstep: bool = True)[source]#

Bases: FieldConfig

SDF Field Config

appearance_embedding_dim: int = 32#

Dimension of appearance embedding

base_res: int = 16#

Base resolution of the encoding

beta_init: float = 0.1#

Init learnable beta value for transformation of sdf to density

bias: float = 0.8#

Sphere size of geometric initialization

divide_factor: float = 2.0#

Normalization factor for multi-resolution grids

features_per_level: int = 2#

Number of features per encoding level

geo_feat_dim: int = 256#

Dimension of geometric feature

geometric_init: bool = True#

Whether to use geometric initialization

hidden_dim: int = 256#

Number of hidden dimension of geometric network

hidden_dim_color: int = 256#

Number of hidden dimension of color network

inside_outside: bool = True#

Whether to revert signed distance value, set to True for indoor scene

log2_hashmap_size: int = 19#

Size of the hash map

max_res: int = 2048#

Maximum resolution of the encoding

num_layers: int = 8#

Number of layers for geometric network

num_layers_color: int = 4#

Number of layers for color network

num_levels: int = 16#

Number of encoding levels

smoothstep: bool = True#

Whether to use the smoothstep function

use_appearance_embedding: bool = False#

Whether to use appearance embedding

use_grid_feature: bool = False#

Whether to use multi-resolution feature grids

use_hash: bool = True#

Whether to use hash encoding

weight_norm: bool = True#

Whether to use weight norm for linear layer

Semantic NeRF#

Semantic NeRF field implementation.

class nerfstudio.fields.semantic_nerf_field.SemanticNerfField(num_semantic_classes: int, position_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, skip_connections: Tuple[int] = (4,))[source]#

Bases: Field

Semantic-NeRF field

Parameters:
  • num_semantic_classes – Number of distinct semantic classes.

  • position_encoding – Position encoder.

  • direction_encoding – Direction encoder.

  • base_mlp_num_layers – Number of layers for base MLP.

  • base_mlp_layer_width – Width of base MLP layers.

  • head_mlp_num_layers – Number of layer for output head MLP.

  • head_mlp_layer_width – Width of output head MLP layers.

  • skip_connections – Where to add skip connection in base MLP.

get_density(ray_samples: RaySamples) Tuple[Tensor, Tensor][source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) Dict[FieldHeadNames, Tensor][source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.

TensoRF#

TensoRF Field

class nerfstudio.fields.tensorf_field.TensoRFField(aabb: Tensor, feature_encoding: Encoding = Identity(), direction_encoding: Encoding = Identity(), density_encoding: Encoding = Identity(), color_encoding: Encoding = Identity(), appearance_dim: int = 27, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, use_sh: bool = False, sh_levels: int = 2)[source]#

Bases: Field

TensoRF Field

forward(ray_samples: RaySamples, compute_normals: bool = False, mask: Optional[Tensor] = None, bg_color: Optional[Tensor] = None) Dict[FieldHeadNames, Tensor][source]#

Evaluates the field at points along the ray.

Parameters:

ray_samples – Samples to evaluate field on.

get_density(ray_samples: RaySamples) Tensor[source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) Tensor[source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.

Vanilla NeRF#

Classic NeRF field

class nerfstudio.fields.vanilla_nerf_field.NeRFField(position_encoding: ~nerfstudio.field_components.encodings.Encoding = Identity(), direction_encoding: ~nerfstudio.field_components.encodings.Encoding = Identity(), base_mlp_num_layers: int = 8, base_mlp_layer_width: int = 256, head_mlp_num_layers: int = 2, head_mlp_layer_width: int = 128, skip_connections: ~typing.Tuple[int] = (4, ), field_heads: ~typing.Optional[~typing.Tuple[~typing.Type[~nerfstudio.field_components.field_heads.FieldHead]]] = (<class 'nerfstudio.field_components.field_heads.RGBFieldHead'>, ), use_integrated_encoding: bool = False, spatial_distortion: ~typing.Optional[~nerfstudio.field_components.spatial_distortions.SpatialDistortion] = None)[source]#

Bases: Field

NeRF Field

Parameters:
  • position_encoding – Position encoder.

  • direction_encoding – Direction encoder.

  • base_mlp_num_layers – Number of layers for base MLP.

  • base_mlp_layer_width – Width of base MLP layers.

  • head_mlp_num_layers – Number of layer for output head MLP.

  • head_mlp_layer_width – Width of output head MLP layers.

  • skip_connections – Where to add skip connection in base MLP.

  • use_integrated_encoding – Used integrated samples as encoding input.

  • spatial_distortion – Spatial distortion.

get_density(ray_samples: RaySamples) Tuple[Tensor, Tensor][source]#

Computes and returns the densities. Returns a tensor of densities and a tensor of features.

Parameters:

ray_samples – Samples locations to compute density.

get_outputs(ray_samples: RaySamples, density_embedding: Optional[Tensor] = None) Dict[FieldHeadNames, Tensor][source]#

Computes and returns the colors. Returns output field values.

Parameters:
  • ray_samples – Samples locations to compute outputs.

  • density_embedding – Density embeddings to condition on.