Standard ray image processing

The fundamental concept behind the Raytrix lightfield API is shown in the following diagram.

A ray image loaded from disc with RxRayLoad() is first of all stored in the Raytrix API DLL. The function RxRayLoad() does not return the ray image itself, but a handle to the ray image. Before any calculations can be performed with the ray image, it has to be copied to the CUDA device, since all operations are performed by CUDA. Copying the ray image to the CUDA device is done by the function RxRayBind(). After this function is called, all ray image processing operations act on the bound ray image, without the need for repeated copying operations of the ray image to the CUDA device.

The diagram shows two example operations on a bound ray image: RxRefocusBasic() to generate a resultant image focused to a particular plane and RxDepthRay() and RxDepthMap() to calculate the scene depth from the ray image. These two algorithms write to specific result images that are held on the CUDA device. All result images are listed in Rx::ApiLF::EImgID. The reason for keeping the result images initially on the CUDA device is, that they can then be processed further by algorithms running on the CUDA device and that they can be displays directly by the graphics card used using OpenGL. For more details on the CUDA/OpenGL interop. see CUDA/OpenGL interop. functions.

The result images can also be copied back to the host device using RxGetImage(). They are then available for further processing by an image processing library of your choice.

If a ray image is unbound it is not deleted internally. That is, it can be bound again, or saved. Make sure to delete ray images that are no longer needed with RxRayDelete(). Otherwise the memory may fill up rather quickly. If you call RxFinalize() all ray images in memory are deleted, all cameras are closed and any memory allocated on the CUDA device is freed. However, you cannot call any API functions except for RxInit() after a call of RxFinalize().

Camera capture ray image processing

Before you can capture images from a Raytrix camera you have to do the following preparations:

Initialize the available camera drivers with RxCamDriverInit().
Register the available cameras with RxCamRegister().
Open a camera with RxCamOpen().
Start the capture process with RxCamStartCapture().

When all of this was successful, images can be captured with RxCamRetrieveImage().

For the ray image processing algorithms to be able to work with captured ray images, the ray image calibration has to be known. Such a calibration contains information on where the image of the central microlens is located in the image, the rotation of the microlens array with respect to the sensor coordinate system and a scaling of the microimages. These parameters not only change for different lenses mounted on a camera, they also vary as the lens focus and/or focal length is changed.

Important: It is important to realize that the effective aperture of a Raytrix camera is fixed. This implies that as the focus of a lens is changed, the aperture setting of the aperture ring has to be adjusted such that the effective aperture is constant. See the RxLive user manual for examples of this.

Since there are different calibration settings for each setting of each lens used, the Raytrix API implements a calibration database that keeps all this data. Particular calibration datasets are selected from the calibration database using the function RxCamGetPar() with parameter IDs RxPar_CamCal_LensIdx and RxPar_CamCal_LensSetIdx which are defined in Rx::ApiLF::EPar::ID. With the first parameter a lens is selected in the calibration database and with the second parameter the lens setting of the currently selected lens is selected. The name and ID of the currently selected lens can be read with RxCamGetPar() and the parameter IDs RxPar_CamCal_LensName and RxPar_CamCal_LensID, respectively. Similarly, the f-stop and focus of the currently selected lens setting can be read with the parameter IDs RxPar_CamCal_LensSetFStop and RxPar_CamCal_LensSetFocus, respectively.

The processing of ray image data captured with a Raytrix camera is shown in the following diagram.

Camera capture image processing

The function RxCamRetrieveImage() captures a camera image into a ray image held by the Raytrix API DLL. This ray image is also automatically copied to the CUDA device. That is, there is no need to call RxRayBind() explicitly. The ray image on the CUDA device can then be used by the various ray image processing algorithms. The result images of the ray image algortihms may be copied back to the host with RxGetImage().

Contents

Standard ray image processing

Camera capture ray image processing