Custom QtVTK Widgets

Overlay Widget

Module to provide a set of VTK renderers that can be used to create an Augmented Reality viewer.

Expected usage:

window = VTKOverlayWindow()
window.add_vtk_models(list)       # list of models. See class hierarchy in sksurgeryvtk/models.
window.add_vtk_actor(actor)       # and/or individual VTK actor.
window.set_camera_matrix(ndarray) # Set 3x3 ndarray of OpenCV camera intrinsic matrix.

while True:

    image = # acquire np.ndarray image some how, e.g. from webcam or USB source.

    window.set_video_image(image)           # We use OpenCV, so this function expects BGR.
    window.set_camera_pose(camera_to_world) # set 4x4 ndarray representing camera pose.

class sksurgeryvtk.widgets.vtk_overlay_window.VTKOverlayWindow(offscreen=False, camera_matrix=None, clipping_range=(1, 1000), zbuffer=False, opencv_style=True, init_pose=False, reset_camera=True, init_widget=True, video_in_layer_0=True, video_in_layer_2=False, layer_2_video_mask=None, use_depth_peeling=True)

Bases: QVTKRenderWindowInteractor

Sets up a VTK Overlay Window that can be used to overlay multiple VTK models on a video stream. Internally, the Window has 5 renderers, 0=backmost, 4=frontmost.

# Layer 0: Video # Layer 1: VTK rendered models - e.g. internal anatomy # Layer 2: Video # Layer 3: VTK rendered models - e.g. external anatomy # Layer 4: VTK rendered text annotations.

The video input should be BGR (like OpenCV provides). You can choose in the constructor whether the video should go to Layer 0 or Layer 2.

If you put video in the Layer 0, and overlay models in Layer 1, you get a simple overlay which may be suitable for things like overlaying calibration points on chessboards, but you will get poor visual coherence for medical AR, as the bright colours of a synthetic overlay will always make the model appear in front of the video, even when you dial back the opacity of the model.

If you put the video in Layer 2, it would obscure the rendered models in Layer 1. But you can apply a mask to the alpha channel setting the alpha to either 0 or 255. If the mask contains say a circle, it will have the effect of showing the video when the alpha channel is 255 and the rendering behind when the alpha channel is 0. So, you can use this to peek inside an organ by rendering the video in Layer 2 with a mask creating a hole, and the internal anatomy in Layer 1. Then you put the external anatomy, e.g. liver surface in Layer 3.

Parameters:

• offscreen – Enable/Disable offscreen rendering.

• camera_matrix – Camera intrinsics matrix.

• clipping_range – Near/Far clipping range.

• zbuffer – If True, will only render zbuffer of main renderer.

• opencv_style – If True, adopts OpenCV camera convention, otherwise OpenGL.

• init_pose – If True, will initialise the camera pose to identity.

• reset_camera – If True, resets camera when a new model is added.

• init_widget – If True we will call self.Initialize and self.Start as part of the init function. Set to false if you’re on Linux.

• video_in_layer_0 – If true, will add video to Layer 0, fully opaque, no masking.

• video_in_layer_2 – If true, will add video to Layer 1. If layer_2_video_mask is present, will mask alpha channel.

add_vtk_actor(actor, layer=1)

Add a vtkActor directly.

Parameters:

• actor – vtkActor

• layer – [1|3|4]. Render layer to add to, default 1.

add_vtk_models(models, layer=1)

Add VTK models to a renderer. Here, a ‘VTK model’ is any object that has an attribute called actor that is a vtkActor. See class hierarchy in sksurgeryvtk/models.

Parameters:

• models – list of VTK models.

• layer – [1|3|4]. Render layer to add to, default 1.

closeEvent(self, event: PySide6.QtGui.QCloseEvent) → None

convert_scene_to_numpy_array()

Convert the current window view to a numpy array.

Return output:

Scene as numpy array

get_background_image_actor(layer=0)

Returns one of the background video image actors.

Parameters:

layer – [0|2]. Index of image actor. Default 0.

get_background_renderer(layer=0)

Returns one of the background video layers.

Parameters:

layer – [0|2]. Index of background image renderer. Default 0.

get_camera_state(layer=1)

Get all the necessary variables to allow the camera view to be restored. For legacy compatibility, this will assume layer 1, like this class was pre-Feb 3rd 2024.

get_foreground_camera(layer=1)

Returns the camera for the foreground vtkRenderer. For legacy compatibility, this will assume layer 1, like this class was pre-Feb 3rd 2024.

Returns:

vtkCamera

get_foreground_renderer(layer=1)

Returns the foreground vtkRenderer. For legacy compatibility, this will assume layer 1, like this class was pre-Feb 3rd 2024.

Returns:

vtkRenderer

get_overlay_renderer()

This returns the top-most layer, where you might put text annotations for example.

resizeEvent(ev)

Ensures that when the window is resized, the background renderer will correctly reposition the camera such that the image fully fills the screen, and if the foreground renderer is calibrated, also updates the projection matrix.

Parameters:

ev – Event

save_scene_to_file(file_name)

Save’s the current screen to file. VTK works in RGB, but OpenCV assumes BGR, so swap the colour space before saving to file. :param file_name: must be compatible with cv2.imwrite()

set_camera_matrix(camera_matrix)

Sets the camera intrinsic matrix from a numpy 3x3 array. :param camera_matrix: numpy 3x3 ndarray containing fx, fy, cx, cy

set_camera_pose(camera_to_world)

Sets the camera position and orientation, from a numpy 4x4 array. :param camera_to_world: camera_to_world transform.

set_camera_state(camera_properties, layer=1)

Set the camera properties to a particular view poisition/angle etc. For legacy compatibility, this will assume layer 1, like this class was pre-Feb 3rd 2024.

set_foreground_camera(camera, layer=1)

Set the foreground camera to track the view in another window. For legacy compatibility, this will assume layer 1, like this class was pre-Feb 3rd 2024.

set_screen(screen)

Link the widget with a particular screen. This is necessary when we have multi-monitor setups.

Parameters:

screen – QScreen object.

set_stereo_left()

Set the render window to left stereo view.

set_stereo_right()

Set the render window to right stereo view.

set_video_image(input_image)

Sets the video image that is used for the background. See also constructor args video_in_layer_0 and video_in_layer_2 which controls in which layer(s) the video image ends up.

Parameters:

input_image – We use OpenCV, so the input image, should be BGR channel order.

set_video_mask(mask_image)

Allows you to store a mask image, for alpha blending with layer 2 video channel.

Parameters:

mask_image – numpy ndarray. Must be single channel, grey-scale, uint8, same size as input video.

staticMetaObject = PySide6.QtCore.QMetaObject("VTKOverlayWindow" inherits "QVTKRenderWindowInteractor": )

Stereo interlaced Widget

Module to provide an interlaced stereo window, designed for driving things like the Storz 3D laparoscope monitor.

class sksurgeryvtk.widgets.vtk_interlaced_stereo_window.VTKStereoInterlacedWindow(offscreen=False, left_camera_matrix=None, right_camera_matrix=None, clipping_range=(1, 10000), init_widget=True)

Bases: QWidget

Class to contain a pair of VTKOverlayWindows, stacked with a QLabel widget containing the resulting interlaced picture.

Parameters:

init_widget – If True we will call self.Initialize and self.Start as part of the init function. Set to false if you’re on Linux.

add_vtk_actor(actor)

Adds a vtkActor to both left and right widgets.

Parameters:

actor – vtkActor

add_vtk_models(models)

Add models to both left and right widgets. Here a model is anything with an attribute called actor that is a vtkActor.

Parameters:

models – vtk_base_model

closeEvent(self, event: PySide6.QtGui.QCloseEvent) → None

paintEvent(ev)

Ensure that the interlaced image is recomputed.

render()

Calls Render on all 3 contained vtk_overlay_windows.

resizeEvent(ev)

Ensure that the interlaced image is recomputed.

save_scene_to_file(file_name)

Writes the currently displayed widget contents to file.

Parameters:

file_name – file name compatible with cv2.imwrite()

set_camera_matrices(left_camera_matrix, right_camera_matrix)

Sets both the left and right camera matrices.

Parameters:

• left_camera_matrix – numpy 3x3 ndarray containing fx, fy, cx, cy

• right_camera_matrix – numpy 3x3 ndarray containing fx, fy, cx, cy

set_camera_poses(left_camera_to_world)

Sets the pose of both the left and right camera. If you haven’t set the left_to_right transform, it will be identity.

Parameters:

left_camera_to_world – 4x4 numpy ndarray, rigid transform

set_current_viewer_index(viewer_index)

Sets the current viewer selection. Defaults to self.default_viewer_ndex.

0 = left 1 = right 2 = interlaced 3 = stacked

Parameters:

viewer_index – index of viewer, as above.

set_left_to_right(left_to_right)

Sets the left_to_right transform (stereo extrinsics).

Parameters:

left_to_right – 4x4 numpy ndarray, rigid transform

set_video_images(left_image, right_image)

Sets both left and right video images. Images must be the same shape, and have an even number of rows.

Parameters:

• left_image – left numpy image

• right_image – right numpy image

Raises:

ValueError, TypeError

set_view_to_interlaced()

Sets the current view to interlaced.

set_view_to_stacked()

Sets the current view to stacked.

staticMetaObject = PySide6.QtCore.QMetaObject("VTKStereoInterlacedWindow" inherits "QWidget": )

Rendering Generator

Module to provide a basic VTK render window for test data generation.

class sksurgeryvtk.widgets.vtk_rendering_generator.VTKRenderingGenerator(models_file, background_image, intrinsic_file, camera_to_world=None, left_to_right=None, offscreen=False, zbuffer=False, gaussian_sigma=0.0, gaussian_window_size=11, clipping_range=(1, 1000), init_widget=True)

Bases: QWidget

Class contains a VTKOverlayWindow and a few extra functions to facilitate rendering loops for generating test data.

Parameters:

• models_file – JSON file describing VTK models, in SNAPPY format

• background_image – RGB image to render in background

• intrinsic_file – [3x3] matrix in text file, in numpy format

• camera_to_world – list of [rx,ry,rz,tx,ty,tz] in degrees/millimetres

• left_to_right – list of [rx,ry,rz,tx,ty,tz] in degrees/millimetres

• offscreen – if true, renders offscreen

• zbuffer – if true, causes VTK to render just the z-buffer

• gaussian_sigma – if non-zero, adds blurring to the rendered image

• gaussian_window_size – window size of OpenCV Gaussian kernel

• clipping_range – VTK clipping range (near, far)

• init_widget – If True we will call self.Initialize and self.Start as part of the init function. Set to false if you’re on Linux.

closeEvent(self, event: PySide6.QtGui.QCloseEvent) → None

get_image()

Returns the rendered image, with post processing like smoothing. :return: numpy ndarray representing rendered image (RGB)

get_masks()

If we want to render masks for test data for DL models for instance, we typically want distinct masks per model object. This method returns a dictionary of new images corresponding to each named model.

If model is shaded, the shading is turned off to get masks, the masks are acquired, and the shading is applied again.

Note: You should ensure self.gaussian_sigma == 0 (the default), and in the .json file.

set_all_model_to_world(model_to_world)

Decomposes the model_to_world string into rx,ry,rx,tx,ty,rz, constructs a 4x4 matrix, and applies it to all models.

Parameters:

model_to_world – [4x4] numpy ndarray, rigid transform

set_clipping_range(minimum, maximum)

Sets the clipping range on the foreground camera.

Parameters:

• minimum – minimum in millimetres

• maximum – maximum in millimetres

set_model_to_worlds(dict_of_transforms)

Given a dictionary of transforms, will iterate by name, and apply the transform to the named object. :param dict_of_transforms: {name, [rx, ry, rz, tx, ty, tz]}

set_smoothing(sigma, window_size)

Sets the Gaussian blur.

Parameters:

• sigma – standard deviation of Gaussian function.

• window_size – sets the window size of Gaussian kernel (pixels).

setup_camera_extrinsics(camera_to_world, left_to_right=None)

Decomposes parameter strings into 6DOF parameters, and sets up camera-to-world and left_to_right for stereo.

Parameters:

• camera_to_world – list of [rx,ry,rz,tx,ty,tz] in degrees/mm

• left_to_right – list of [rx,ry,rz,tx,ty,tz] in degrees/mm

setup_intrinsics()

Set the intrinsics of the foreground vtkCamera.

staticMetaObject = PySide6.QtCore.QMetaObject("VTKRenderingGenerator" inherits "QWidget": )

Reslice Widget

Module to show slice views of volumetric data.

class sksurgeryvtk.widgets.vtk_reslice_widget.MouseWheelSliceViewer(input_data)

Bases: VTKSliceViewer

Orthogonal slice viewer using mouse wheel to control slice position.

Example usage:

qApp = QtWidgets.QApplication([]) input_data = ‘tests/data/dicom/LegoPhantom_10slices’

slice_viewer = MouseWheelSliceViewer(input_data) slice_viewer.start() qApp.exec_()

start()

Start a timer which will update the 3D view.

staticMetaObject = PySide6.QtCore.QMetaObject("MouseWheelSliceViewer" inherits "VTKSliceViewer": )

update_fourth_panel()

Update 3D view.

class sksurgeryvtk.widgets.vtk_reslice_widget.TrackedSliceViewer(input_data, tracker)

Bases: VTKSliceViewer

Orthogonal slice viewer combined with tracker to control slice position. :param input_data: Path to file/folder containing volume data :param tracker: scikit-surgery tracker object,

used to control slice positions.

Example usage:

qApp = QtWidgets.QApplication([]) input_data = ‘tests/data/dicom/LegoPhantom_10slices’ tracker = ArUcoTracker()

slice_viewer = MouseWheelSliceViewer(input_data, tracker) slice_viewer.start() qApp.exec_()

start()

Show the overlay widget and set a timer running

staticMetaObject = PySide6.QtCore.QMetaObject("TrackedSliceViewer" inherits "VTKSliceViewer": )

update_position()

Get position from tracker and use this to set slice positions.

class sksurgeryvtk.widgets.vtk_reslice_widget.VTKResliceWidget(reader, axis, parent)

Bases: QVTKRenderWindowInteractor

Widget to show a single slice of Volumetric Data. :param reader: vtkReader class e.g. DICOM/Niftii/gipl :param axis: x/y/z axis selection :param parent: parent QWidget.

get_slice_position()

Return the current slice position.

on_mouse_wheel_backward(obj, event)

Callback to change slice position using mouse wheel.

on_mouse_wheel_forward(obj, event)

Callback to change slice position using mouse wheel.

reset_position()

Set slice position to the middle of the axis.

set_lookup_table_min_max(min, max)

Set the minimum/maximum values for the VTK lookup table i.e. change displayed range of intensity values.

set_mouse_wheel_callbacks()

Add callbacks for scroll events.

set_slice_position_mm(pos)

Set the slice position in the volume in mm

set_slice_position_pixels(pos)

Set the slice position in the volume in pixels

staticMetaObject = PySide6.QtCore.QMetaObject("VTKResliceWidget" inherits "QVTKRenderWindowInteractor": )

class sksurgeryvtk.widgets.vtk_reslice_widget.VTKSliceViewer(input_data)

Bases: QWidget

Othrogonal slice viewer showing Axial/Sagittal/Coronal views :param input_data: path to volume data

reset_slice_positions()

Set slcie positions to some default values.

set_lookup_table_min_max(min, max)

Set lookup table min/max for all slice views

staticMetaObject = PySide6.QtCore.QMetaObject("VTKSliceViewer" inherits "QWidget": )

update_slice_positions_mm(x_pos, y_pos, z_pos)

Set the slice positions for each view. :param x: slice 1 position :param y: slice 2 position :param z: slice 3 position

update_slice_positions_pixels(x_pos, y_pos, z_pos)

Set the slice positions for each view. :param x: slice 1 position :param y: slice 2 position :param z: slice 3 position

VTK Model Data

Base Model

Base class to provide a base class definition of what a ‘VTK model’ is. In the context of this project, at this current moment in time, its an object that has a member variable called ‘actor’ that is a vtkActor.

class sksurgeryvtk.models.vtk_base_model.VTKBaseModel(colour, visibility=True, opacity=1.0, pickable=True, outline=False)

Bases: VTKBaseActor

Defines a base class for ‘VTK Models’ which are objects that contain at least one vtkActor. From v1.1 we can optionally contain an additional outline rendering vtkActor This class enables you to set the colour, visibility and opacity. Note that this colour property is set on the actor. It is possible for various VTK implementations to ignore this. For example a point set could store an RGB tuple for each point, so when rendered, the overall colour property is effectively ignored. However, the property has been kept at this base class level for simplicity.

get_name()

Returns the name of the model.

Returns:

str, the name, which can be None if not yet set.

get_outline()

Returns the outline flag.

get_outline_actor(active_camera)

Sets up the outline renderer. and returns the outline actor so you can add it to your renderer Before doing this self.actor should have been set up with a mapper and we need a camera to know where we’re projecting from.

Parameters:

active_camera – the vtk camera. Use vtk_overlay.foreground_renderer.GetActiveCamera()

:returns the outline actor

get_user_matrix()

Getter for vtkActor UserMatrix. :return: vtkMatrix4x4

set_name(name)

Sets the name.

Parameters:

name – str containing a name

Raises:

TypeError if not string, ValueError if empty

set_outline(outline)

Enables the user to set the outline rendering flag.

Parameters:

outline –

Raises:

TypeError if not a boolean

set_user_matrix(matrix)

Sets the vtkActor UserMatrix. This simply tells the graphics pipeline to move/translate/rotate the actor. It does not transform the original data.

Parameters:

matrix – vtkMatrix4x4

Surface Models

VTK pipeline to represent a surface model via a vtkPolyData.

class sksurgeryvtk.models.vtk_surface_model.VTKSurfaceModel(filename, colour, visibility=True, opacity=1.0, pickable=True, outline=False)

Bases: VTKBaseModel

Class to represent a VTK surface model. Normally read from a file, but could be created on the fly.

get_model_transform()

Gets the model to world transform. :return: vtkMatrix4x4

get_no_shading()

Returns whether or not this model is rendered with or without shading. :return: bool

get_normals_as_numpy()

Returns the vtkPolyData point normals as a numpy array.

Returns:

nx3 numpy ndarray

get_number_of_points()

Returns the number of points in the vtkPoylData. :return: unsigned int

get_points_as_numpy()

Returns the vtkPolyData points as a numpy array. :return: nx3 numpy ndarray

get_source_file()

Returns the filename that the model was loaded from, or empty string if the VTKSurfaceModel was not made from a file.

:return:str filename

get_vtk_source_data() → vtkPolyData

Return original vtk poly data for this object

Returns:

vtkPolyData

Return type:

vtk.vtkPolyData

set_model_transform(matrix)

Sets the model to world transform onto a vtkPolyDataFilter. This enables all the points and point data to be transformed according to a vtkMatrix4x4 similarity transform.

Parameters:

matrix – vtkMatrix4x4

set_no_shading(no_shading: bool)

Turns off/on all shading, so you can generate masks, with solid blocks of colour. Note: Even though I’m tempted to call this flat shading, you can’t because flat shading is something different. So we have to call it “no shading”.

Parameters:

no_shading – if true, outputs solid blocks of colour

set_texture(filename)

Sets an image from a file as a texture for the model. :param filename: :return:

Module to load VTK surfaces using dictionary from ConfigurationManager.

class sksurgeryvtk.models.surface_model_loader.SurfaceModelLoader(data, directory_prefix=None)

Bases: object

Class to load VTK surface models and (optionally) associate them with vtkAssembly’s. Surfaces should be defined in a .json file and loaded for example using sksurgerycore.ConfigurationManager.

Surfaces have format:

"surfaces": {
    "tumor": {
        "file": "path/to/model/tumor.vtk",
        "colour": [255, 0, 0],
        "opacity": 0.5,
        "visibility": true,
        "pickable": true,
        "toggleable": true,
        "texture": "path/to/texture/image.png",
        "outline": false
    }

Assemblies have format:

"assemblies": {
    "whole model": ["part1", "part2", "part3"]
}

get_assembly(name)

Fetches a vtkAssembly using the name.

Parameters:

name – name of the assembly, as string

Returns:

vtkAssembly

get_assembly_names()

Returns the set of valid assembly names.

Returns:

keys from self.named_assemblies

get_surface_model(name)

Fetches a VTKSurfaceModel using the name.

Parameters:

name – name of the model

Returns:

VTKSurfaceModel

get_surface_model_names()

Returns the set of valid model names.

Returns:

keys from self.named_surfaces

get_surface_models()

Convenience method, to get all models.

Useful for unit testing for example.

Returns:

list of VTKSurfaceModel

Module to load VTK surfaces.

class sksurgeryvtk.models.vtk_surface_model_directory_loader.VTKSurfaceModelDirectoryLoader(directory_name, defaults_file=None)

Bases: object

Class to load all VTK surface models in a directory.

get_model_colours(directory)

Load colours for each model from a .txt file in the model directory.

get_models(directory_name)

Loads models from the given directory.

Parameters:

directory_name – string, readable directory name.

Raises:

TypeError, ValueError, RuntimeError

Unstructured Grid Model

VTK pipeline to represent a vtkUnstructredGrid.

class sksurgeryvtk.models.vtk_grid_model.VTKUnstructuredGridModel(filename, colour=[1.0, 0.0, 0.0], visibility=True, opacity=1.0, pickable=True)

Bases: VTKBaseModel

Class to represent a VTK grid model read from a file.

get_cell_array() → ndarray

Return the cell data array as a numpy array

Returns:

Cell data scalars

Return type:

np.ndarray

get_cell_array_bounds() → Tuple[float, float]

Return min/max values of cell array data.

Returns:

Min, Max

Return type:

float, float

get_source_file()

Returns the filename that the model was loaded from, or empty string if the VTKSurfaceModel was not made from a file.

:return:str filename

set_cell_array(data: ndarray)

Set the cell data in the grid.

Parameters:

data (np.ndarray) – Numpy array of cell data points

threshold_between(lower: float, upper: float)

Set upper/lower limits to use for vtkThreshold

Parameters:

• lower (float) – Lower limit

• upper (float) – Upper limit

Image Model

VTK pipeline to represent an image with a vtkImageActor.

class sksurgeryvtk.models.vtk_image_model.VTKImageModel(filename, visibility=True, opacity=1.0)

Bases: VTKBaseModel

Class to represent a VTK image model. Normally read from a file, but could be created on the fly.

Point Model

VTK pipeline to represent a point model via a vtkPolyData with a separate (RGB) component for each point, such that each point is rendered with the correct colour. Note that this model is designed to have a fixed number of points. If you want varying number of points for each render pass, you should consider another way of doing this.

class sksurgeryvtk.models.vtk_point_model.VTKPointModel(points, colours, visibility=True, opacity=1.0)

Bases: VTKBaseModel

Class to represent a VTK point model. Note, that if

get_number_of_points()

Returns the number of points (hence vertices) in the model. :return: number of points

get_point_size()

Returns the current point size in pixels. :return: size

set_point_size(size)

Sets the size of each point in pixels.

Geometric Primitives

VTK pipeline to represent a set of points, as sphere glyphs.

class sksurgeryvtk.models.vtk_sphere_model.VTKSphereModel(points, radius, colour=(1.0, 1.0, 1.0), visibility=True, opacity=1.0, pickable=True, resolution=12)

Bases: VTKBaseModel

Class to represent a set of points as sphere glyphs (one sphere per point).

VTK pipeline to represent a surface model via a vtkPolyData.

class sksurgeryvtk.models.vtk_cylinder_model.VTKCylinderModel(height=10.0, radius=3.0, colour=(1.0, 0.0, 0.0), name='cylinder', angle=90.0, orientation=(1.0, 0.0, 0.0), resolution=88, visibility=True, opacity=1.0)

Bases: VTKSurfaceModel

Class to create a VTK surface model of a cylinder.

Camera Utilities

Functions to setup a VTK camera to match the OpenCV calibrated camera.

sksurgeryvtk.camera.vtk_camera_model.compute_projection_matrix(width, height, f_x, f_y, c_x, c_y, near, far)

Computes the OpenGL projection matrix.

Thanks to: Andrew Straw.

whose method was also implemented in: NifTK.

and it appears it should also be available in: VTK.

Note: If you use this method, the display will look ok, but as of VTK 8.1.0, it won’t work with vtkWindowToImageFilter, as the window to image filter tries to render the image in tiles. This requires instantiating temporary new vtkCamera, and the vtkCamera copy constructor, shallow copy and deep copy do not actually copy the UseExplicitProjectionTransformMatrixOn or ExplicitProjectionTransformMatrix.

Parameters:

• width – image width in pixels

• height – image height in pixels

• f_x – focal length in x direction, (K_00)

• f_y – focal length in y direction, (K_11)

• c_x – principal point x coordinate, (K_02)

• c_y – principal point y coordinate, (K_12)

• near – near clipping distance in world coordinate frame units (mm)

• far – far clipping distance in world coordinate frame units (mm)

Returns:

vtkMatrix4x4 containing a 4x4 projection matrix

sksurgeryvtk.camera.vtk_camera_model.compute_right_camera_pose(left_camera_to_world, left_to_right)

Returns the right_camera_to_world, computed from the combination of left_camera_to_world, and left_to_right.

Parameters:

• left_camera_to_world – 4x4 numpy ndarray representing rigid transform

• left_to_right – 4x4 numpy ndarray representing rigid transform

Returns:

right_camera_to_world as 4x4 numpy ndarray

sksurgeryvtk.camera.vtk_camera_model.compute_scissor(window_width, window_height, image_width, image_height, aspect_ratio)

Used on vtkCamera when you are trying to set the viewport to only render to a part of the total window size. For example, this occurs when you have calibrated a video camera using OpenCV, on images of 1920 x 1080, and then you are displaying in a VTK window that is twice as wide/high.

Parameters:

• window_width – in pixels

• window_height – in pixels

• image_width – in pixels

• image_height – in pixels

• aspect_ratio – relative physical size of pixels, as x/y.

Returns:

scissor_x, scissor_y, scissor_width, scissor_height in pixels

sksurgeryvtk.camera.vtk_camera_model.compute_viewport(window_width, window_height, scissor_x, scissor_y, scissor_width, scissor_height)

Used on vtkCamera when you are trying to set the viewport to only render to a part of the total window size. For example, this occurs when you have calibrated a video camera using OpenCV, on images of 1920 x 1080, and then you are displaying in a VTK window that is twice as wide/high.

Parameters:

• window_width – in pixels

• window_height – in pixels

• scissor_x – output from compute_scissor

• scissor_y – output from compute_scissor

• scissor_width – output from compute_scissor

• scissor_height – output from compute_scissor

Returns:

x_min, y_min, x_max, y_max as normalised viewport coordinates

sksurgeryvtk.camera.vtk_camera_model.set_camera_intrinsics(vtk_renderer, vtk_camera, width, height, f_x, f_y, c_x, c_y, near, far)

Used to setup a vtkCamera according to OpenCV conventions.

Thanks to: benoitrosa

Parameters:

• vtk_renderer – vtkRenderer

• vtk_camera – vtkCamera

• width – image width in pixels

• height – image height in pixels

• f_x – focal length in x direction, (K_00)

• f_y – focal length in y direction, (K_11)

• c_x – principal point x coordinate, (K_02)

• c_y – principal point y coordinate, (K_12)

• near – near clipping distance in world coordinate frame units (mm).

• far – far clipping distance in world coordinate frame units (mm).

sksurgeryvtk.camera.vtk_camera_model.set_camera_pose(vtk_camera, vtk_matrix, opencv_style=True)

Sets the camera position and orientation from a camera to world matrix.

If opencv_style is False, the camera defaults to the origin, facing along the -z axis, with +y being up.

If opencv_style is True (default for legacy compatibility), the camera defaults to the origin, facing along the +z axis, with +y being down. This is more in-line with Opencv. So, if you are calibrating with OpenCV, and want to use those extrinsic matrices to set the pose, then you want this option.

Parameters:

• vtk_camera – a vtkCamera

• vtk_matrix – a vtkMatrix4x4 representing the camera to world.

• opencv_style – If True uses OpenCV (+z), otherwise OpenGL (-z)

Text Overlay

Classes to implement text overlay. Includes Corner Annotation, Large Centered Text and generic text overlay.

class sksurgeryvtk.text.text_overlay.VTKCornerAnnotation

Bases: object

Wrapper for vtkCornerAnnotaiton class.

get_text()

Returns the current list of text annotations :return: [bottom-left, bottom-right, top-left, top-right]

set_text(text_list)

Set the text in each of the four corners

Parameters:

text_list (List of 4 strings.) – Text to display. [bottom-left, bottom-right, top-left, top-right].

set_text_on_bottom_left(text)

Set the text on the bottom-left corner.

Parameters:

text – Text to display.

set_text_on_bottom_right(text)

Set the text on the bottom-right corner.

Parameters:

text – Text to display.

set_text_on_top_left(text)

Set the text on the top-left corner.

Parameters:

text – Text to display.

set_text_on_top_right(text)

Set the text on the top-right corner.

Parameters:

text – Text to display.

validate_input(text_list)

Check that the text_list input is a list of four strings.

Parameters:

text_list – input to check.

class sksurgeryvtk.text.text_overlay.VTKLargeTextCentreOfScreen(text)

Bases: VTKTextBase

Display large text in the centre of the screen. Useful for error messages/warnings etc.

Parameters:

text – text to display.

calculate_text_size(_obj_unused, _ev_unused)

Calculate the position and size of the text. Text should span the central half (x & y) of the window.

set_parent_window(parent_window)

Attach text to a particular window.

Parameters:

parent_window – VTKOverlayWindow that message will be displayed in.

class sksurgeryvtk.text.text_overlay.VTKText(text, x, y, font_size=24, colour=(1.0, 0, 0))

Bases: VTKTextBase

VTKText object that can be placed following a left click event. Text will rescale if the window resizes, to try and maintain relative positioning.

Parameters:

• text – text to display.

• x – x position (pixels)

• y – y position (pixels)

• font_size – Font size

• colour – Colour, RGB tuple

calculate_relative_position_in_window()

Calculate position relative to the middle of the screen. Can then be used to re-set the position if the window is resized.

callback_update_position_in_window(_obj_unused, _ev_unused)

Update position, maintaing relative distance to the centre of the background image.

set_parent_window(parent_window)

Link the object to a VTKOverlayWindow and set up callbacks.

Parameters:

parent_window – VTKOverlayWindow

class sksurgeryvtk.text.text_overlay.VTKTextBase

Bases: object

Wrapper around vtkTextActor class to set position, colour, size etc.

set_colour(r, g, b)

Set the text colour. :param r: Red (0.0 - 1.0) :param g: Green (0.0 - 1.0) :param b: Blue (0.0 - 1.0)

set_font_size(size)

Set the font size. :param size: size in points

set_text_position(x, y)

Set the x,y coordinates of the text (bottom-left) :param x: x location in pixels :param y: y locaiton in pixels

set_text_string(text)

Set the text string. :param text: text to display.

validate_text_input(text)

Check text input is a valid string. :param text: Input to validate.

validate_x_y_inputs(x, y)

Check that coordinate inputs are valid. :param x: x location. :param y: y location

Misc Utilities

Matrix Utilities

Any useful little utilities to do with matrices.

sksurgeryvtk.utils.matrix_utils.calculate_l2r_matrix(left_extrinsics: ndarray, right_extrinsics: ndarray) → ndarray

Return the left to right transformation matrix:

l2r = R * L^-1

sksurgeryvtk.utils.matrix_utils.create_matrix_from_list(params, is_in_radians=False)

Generates a 4x4 numpy ndarray from a list of rx,ry,rz,tx,ty,tz in degrees, millimetres.

This is designed to match VTK. VTK states that vtkProp3D uses ‘Orientation is specified as X,Y and Z rotations in that order, but they are performed as RotateZ, RotateX, and finally RotateY. However vtkTransform by default uses pre-multiplication. So, in mathematical notation, this would be written as

[Output Point] = [RotateZ][RotateX][RotateY][Input Point]

which, if you read the maths expression from right to left, would actually be termed RotateY, then RotateX, then RotateZ.

The function in scikit-surgerycore called construct_rotm_from_euler takes an input string, e.g. ‘zxy’ and follows mathematical notation. So, ‘zxy’ means RotateY, RotateX, RotateZ in that order, reading from right to left, and so matches VTK.

Furthermore, the construct_rotm_from_euler function in scikit-surgerycore expectes the user to pass the parameters in, in the order specified in the provided string.

:param params list of exactly 6 numbers. :param is_in_radians: True if radians, False otherwise, default is False

sksurgeryvtk.utils.matrix_utils.create_matrix_from_string(parameter_string, is_in_radians=False)

Generates a 4x4 numpy ndarray from a comma separated string of the format rx,ry,rz,tx,ty,tz in degrees, millimetres.

Parameters:

• parameter_string – rx,ry,rz,tx,ty,tz in degrees/millimetres

• is_in_radians – True if radians, False otherwise, default is False

Returns:

4x4 rigid body transform

sksurgeryvtk.utils.matrix_utils.create_numpy_matrix_from_vtk(matrix)

Returns a new numpy 4x4 matrix from a vtkMatrix4x4.

sksurgeryvtk.utils.matrix_utils.create_vtk_matrix_from_numpy(array)

Return a new vtkMatrix4x4 from a numpy array.

sksurgeryvtk.utils.matrix_utils.get_l2r_smartliver_format(l2r_matrix: ndarray) → ndarray

Convert 4x4 left to right matrix to smartliver l2r format:

R1 R2 R3 R4 R5 R6 R7 R8 R9 T1 T2 T3

sksurgeryvtk.utils.matrix_utils.validate_vtk_matrix_4x4(matrix)

Checks that a matrix is a vtkMatrix4x4. :param matrix: vtkMatrix4x4 :raises TypeError :return: True

Projection Utilities

Any useful little utilities to do with projecting 3D to 2D.

sksurgeryvtk.utils.projection_utils.compute_rms_error(model_points, image_points, renderer, scale_x, scale_y, image_height)

Mainly for unit testing. Computes rms error between projected model points, and image points.

Parameters:

• model_points – nx3 numpy array of 3D points

• image_points – nx2 numpy array of 2D expected points

• renderer – vtkRenderer

• scale_x – scale factor for x

• scale_y – scale factor for y

• image_height – image height

sksurgeryvtk.utils.projection_utils.project_facing_points(points, normals, camera_to_world, camera_matrix, distortion=None, upper_cos_theta=0)

Projects 3D points that face the camera to 2D pixels.

This assumes: Camera direction is a unit vector from the camera, towards focal point. Surface Normal is a unit vector pointing out from the surface. Vectors are not checked for unit length.

Parameters:

• points – nx3 ndarray representing 3D points, typically in millimetres

• normals – nx3 ndarray representing unit normals for the same points

• camera_to_world – 4x4 ndarray representing camera to world transform

• camera_matrix – 3x3 ndarray representing OpenCV camera intrinsics

• distortion – 1x4,5 etc. OpenCV distortion parameters

• upper_cos_theta – upper limit for cos theta, angle between normal and viewing direction, where cos theta is normally -1 to 0.

Raises:

ValueError, TypeError:

Returns:

projected_facing_points_2d

sksurgeryvtk.utils.projection_utils.project_points(points, camera_to_world, camera_matrix, distortion=None)

Projects all 3D points to 2D, using OpenCV cv2.projectPoints().

Parameters:

• points – nx3 ndarray representing 3D points, typically in millimetres

• camera_to_world – 4x4 ndarray representing camera to world transform

• camera_matrix – 3x3 ndarray representing OpenCV camera intrinsics

• distortion – 1x4,5 etc. OpenCV distortion parameters

Raises:

ValueError, TypeError:

Returns:

nx2 ndarray representing 2D points, typically in pixels

Polydata Utilities

Utilities for operations on vtk polydata

sksurgeryvtk.utils.polydata_utils.check_overlapping_bounds(polydata_0, polydata_1)

Checks whether two polydata have overlapping bounds

Parameters:

• polydata_0 – vtkPolyData representing a 3D mesh

• polydata_1 – vtkPolyData representing a 3D mesh

:return : True if bounding boxes overlap, False otherwise

sksurgeryvtk.utils.polydata_utils.two_polydata_dice(polydata_0, polydata_1)

Calculates the DICE score for two polydata. Will probably struggle with complex topologies, but should be fine for vaguely spherical shape. This function uses vtk.vtkMassProperties() so does not convert polydata to image data

Parameters:

• polydata_0 – vtkPolyData representing a 3D mesh

• polydata_1 – vtkPolyData representing a 3D mesh

Return dice:

The DICE score

Return volume_0:

The enclosed volume of polydata_0

Return volume_1:

The enclosed volume of polydata_1

Return volume_01:

The enclosed volume of the intersection

Voxelisation & Distance Fields

Re-impelmentaiton of voxeilsation code from https://gitlab.com/nct_tso_public/Volume2SurfaceCNN

sksurgeryvtk.models.voxelise.applyTransformation(dataset, tf)

Apply a transformation to each data array stored in vtk object.

Parameters:

• dataset – Vtk object containing array(s)

• tf (vtk.vtkTransform) – Transform

sksurgeryvtk.models.voxelise.apply_displacement_to_mesh(mesh: vtkDataObject | str, field: vtkStructuredGrid | str, save_mesh: bool | str = False, disp_array_name: str = 'estimatedDisplacement')

Apply a displacement field to a mesh. The displacement field is stored as an array within a vtkStructuredGrid.

Parameters:

• mesh (Union[vtk.vtkDataObject, str]) – Mesh to deform, can either be path to file or vtk object.

• field (Union[vtk.vtkStructuredGrid, str]) – Grid containing displacement field, can either be path to file or vtk object.

• save_mesh (Union[bool, str], optional) – If a file name is passed, the deformed mesh is saved to disk, defaults to False

• disp_array_name (str, optional) – Name of array within vtkStructuredGrid containing the displacement field, defaults to ‘estimatedDisplacement’

Returns:

Displaced mesh

Return type:

vtk.vtkPolyData

sksurgeryvtk.models.voxelise.createGrid(total_size: float, grid_elements: int)

Returns a vtkStrucutredGrid.

Parameters:

• total_size – Total size of the grid i.e. How long is each dimension. Each indivdual element has size equal to total_size/grid_dims

• grid_dims (int) – Number of grid points in x/y/z

Returns:

grid

Return type:

vtkStructuredGrid

sksurgeryvtk.models.voxelise.distanceField(surfaceMesh, targetGrid, targetArrayName: str, signed=False)

Create a distance field between a vtkStructuredGrid and a surface.

Parameters:

• surfaceMesh – Outer polygonal surface

• targetGrid (vtk.vtkStructuredGrid) – Grid array of points

• targetArrayName (str) – The distance field values will be stored in the target grid, with this array name.

• signed (bool, optional) – Signed/unsigned distance field, defaults to False (unsigned)

sksurgeryvtk.models.voxelise.distanceFieldFromCloud(surfaceCloud, targetGrid, targetArrayName)

Create a distance field between a vtkStructuredGrid and a point cloud.

Parameters:

• surfaceMesh – Pointcloud of surface

• targetGrid (vtk.vtkStructuredGrid) – Grid array of points

• targetArrayName – The distance field values will be stored in the target grid, with this array name.

sksurgeryvtk.models.voxelise.extractSurface(inputMesh)

Extract surface of a mesh.

sksurgeryvtk.models.voxelise.extract_array_from_grid(input_grid: vtkStructuredGrid, array_name: str) → ndarray

Read an array from a vtkStructuredGrid object

Parameters:

• input_grid (vtk.vtkStructuredGrid) – Input data grid

• array_name (str) – Array to extract from grid

Returns:

Extracted array

Return type:

np.ndarray

sksurgeryvtk.models.voxelise.extract_array_from_grid_file(input_grid_file: str, array_name: str) → ndarray

Read an array from vtkStructuredGrid file

Parameters:

• input_grid_file (str) – Input file, should be a vtkStructuredGrid file

• array_name (str) – Array to extract from grid

Returns:

Extracted array

Return type:

np.ndarray

sksurgeryvtk.models.voxelise.extract_surfaces_for_v2snet(input_grid: vtkStructuredGrid) → Tuple[ndarray, ndarray]

Conveience function to extract the pre and intraoperative surfaces, to pass to V2SNet.

Parameters:

input_grid (vtk.vtkStructuredGrid) – Grid containing pre and intraoperative surfaces

Returns:

pre and intraoperative surfaces as numpy arrays

Return type:

Tuple[np.ndarray, np.ndarray]

sksurgeryvtk.models.voxelise.loadTransformationMatrix(grid)

Extract a transformation matrix fom a vtk grid array.

sksurgeryvtk.models.voxelise.load_points_from_file(filename)

Extract vtk mesh from input file. :returns: Vtk mesh.

sksurgeryvtk.models.voxelise.load_structured_grid(input_file: str)

Load vtkStructuredGrid from file

Parameters:

input_file (str) – Path to vtk structured grid file

Raises:

TypeError –

Returns:

Loaded grid

Return type:

vtk.vtkStructuredGrid

sksurgeryvtk.models.voxelise.save_displacement_array_in_grid(array: ndarray, out_grid: vtkStructuredGrid | str, array_name: str = 'estimatedDisplacement')

Save numpy data as an array within a vtkStructuredGrid. Mainly used for storing calculated displacement field.

Parameters:

• array (np.ndarray) – Numpy array

• out_grid (Union[vtk.vtkStructuredGrid, str]) – Grid in which to store array

• array_name (str, optional) – Array name, defaults to “estimatedDisplacement”

sksurgeryvtk.models.voxelise.storeTransformationMatrix(grid, tf)

Store a transformation matrix inside a vtk grid array.

sksurgeryvtk.models.voxelise.unstructuredGridToPolyData(ug)

Convert vtk unstructured grid to vtk poly data.

sksurgeryvtk.models.voxelise.voxelise(input_mesh: ndarray | vtkDataObject | str, output_grid: vtkStructuredGrid | str | None = None, array_name: str = '', size: float = 0.3, grid_elements: int = 64, move_input: float | None = None, center: bool = False, scale_input: float | None = None, reuse_transform: bool = False, signed_df: bool = True)

Creates a voxelised distance field, stores it in a vtkStructuredGrid, optimally writes to disk.

Parameters:

• input_mesh (Union[np.ndarray, str) – Input mesh/points. Can be path to model file, or numpy array. Units of mesh should be in metres.

• output_grid (Union[vtk.vtkStructuredGrid, str], optional) – Either a vtkStrucutredGrid object, or a file that contains one (or will be created). If not specified, a grid will be created.

• array_name (str, optional) – Name of array in which to store distance field, if not specified, defaults to preoperativeSurface for if signed_df = True, else intraoperativeSurface

• size (float, optional) – Grid size, defaults to 0.3

• grid_elements – Number of x/y/z elements in grid, defaults to 64 :type grid_elements: int, optional

• move_input (float, optional) – Move the input before transforming to distance field (movement is applied before scaling! defaults to None

• center (bool, optional) – Center the data around the origin. defaults to False

• scale_input (float, optional) – Scale the input before transforming to distance field (movement is applied before scaling!). Input is expected to be in metres, if it is in mm, set scale_input to 0.001 defaults to None

• reuse_transform (bool, optional) – Reuse transformation already stored in the grid. Use this if you want to center mesh 1 and then apply the same transformation to mesh 2. Mutually exclusive with center, scale_input and move_input. defaults to False

• signed_df (bool, optional) – Calcualte signed or unsigned distance field. defaults to True

Return grid:

Grid containing distance field.

Return type:

vtk.vtkStructuredGrid

sksurgeryvtk.models.voxelise.write_grid_to_file(grid: vtkStructuredGrid, output_grid: str)

Write vtkStructuredGrid to file

Parameters:

• grid (vtk.vtkStructuredGrid) – Grid to write

• output_grid (str) – File path