Chapter 9. Using Optional Video Texture Mapping
Prev |  | Next

## Chapter 9. Using Optional Video Texture Mapping

The Video Library for Indigo2 IMPACT Video contains a texture node for capturing video streams into the texture memory on Indigo2 IMPACT graphics. Once captured, a video field can be used as a texture just as if it were an image loaded into texture memory through function calls to the OpenGL graphics library. [3]

The texture node, VL_TEXTURE, is a software abstraction for the texture map interface hardware on the Indigo2 IMPACT Video optional Color Space Converter and Video Texture Interface daughter board. This hardware converts YUV 4:2:2 (for dual-link, YUVA 4:2:2:4) video input to 8-bit RGB (for dual-link, RGBA) format and transfers it to the texture RAM (TRAM) on the Indigo2 IMPACT graphics board.

The texture map interface hardware has two inputs, link A and link B.

- For single-link transfers, you can specify which input to send to TRAM.

- For dual-link transfers, the RGB data is extracted from link A, and the alpha data is extracted from link B.

The texture-map interface hardware on the optional daughter board also has a scaler chip for extracting a rectangular subregion of the input video field and shrinking it either horizontally or vertically before it is transferred to TRAM.

Finally, the texture-map interface hardware also contains a real-time mipmap generator, which can create and transfer the levels of detail necessary to display the video field in mipmap mode on the graphics subsystem (see the OpenGL Programming Guide for a discussion of mipmapping).

Double-buffered video texture applications require 4 MB of TRAM, which you can obtain by installing the Texture Upgrade to 4 MB of Texture Memory for Indigo2 IMPACT graphics.

Figure 9-1 diagrams the hardware configuration that supports the texture node.

Figure 9-1. Video Texture-Mapping Hardware Configuration Block Diagram

This chapter covers the following topics:

- “Performing Video Texture Mapping”

- “Controls for Video Texture Mapping”

- “OpenGL Functions for Video Texture Mapping”

The chapter concludes with a description of the example program `vidtotex.c`.

## Performing Video Texture Mapping

Initially, you use VL calls to create a path between a source node and the texture drain node. Typically, the source node is a video input node. The texture drain node supports both single-link and dual-link transfers.

- For single-link transfers, use the VL_MGV_TEXTURE_INPUT_LINK control to specify the input link that supplies the data sent to TRAM. In addition, you can use the VL_MGV_TEXTURE_AUTOSWAP control to make the input link selection toggle automatically after each capture into TRAM. This feature is useful for applications that require two live video textures simultaneously.

- For dual-link transfers, an RGBA stream is sent to TRAM. In this mode, the RGB data is extracted from link A and the alpha data is extracted from link B.

Once the path is set up, use the `vlBeginTransfer()` routine. At this point, data is being transferred from the video source to the texture drain. However, to capture this data into the TRAM on the Indigo2 IMPACT graphics board, you must explicitly tell OpenGL to do it; otherwise the data is simply lost.

The OpenGL function `glCopyTexSubImage2DEXT()` captures into TRAM the data that the VL is transferring. To specify to this function that a video transfer is occurring, the GLX read drawable must be set to a video input stream. The function `glXCreateGLXVideoSource()` creates a handle for a video input stream. This handle can then be passed as the read-drawable parameter to the function `glXMakeCurrentReadSGI()`. Once this handle is passed, each call to `glCopyTexSubImage2DEXT()` begins a capture into TRAM of the data that is being sent to the VL texture drain node. It is important to realize that this function begins a capture into TRAM and then returns to the caller. Thus, the caller can continue executing while the video field is being captured.

To use the simplest example, you create a loop that calls `glCopyTexSubImage2DEXT()`, and then draw with the captured video field. To display completely drawn fields, use double-buffered drawing. Even if the drawing routine is very fast, however, this implementation can capture only every other video field that is being sent to the texture node. Although the framebuffer used for drawing is double buffered, the texture buffer itself is not. Any attempt to draw with the single-buffered texture causes the OpenGL draw routine to wait for the video field to finish loading into TRAM. When this load finishes, the drawing begins. However, the video field being sent to the texture node while the drawing is occurring is not captured.

The solution to this problem is to double-buffer the TRAM loads. Then, while a video field is being captured into one of the texture buffers, the other one can be used for drawing. This overlapping of loading one video field into TRAM at the same time that another one is being used for drawing enables you to do real-time video texturing. You can use the OpenGL function `glHint()` to enable double-buffer texture loads on an Indigo2 IMPACT graphics system that has enough TRAM to support it.

| Note: Double-buffered video texture applications require 4 MB of TRAM, which you can obtain by installing the Texture Upgrade to 4 MB of Texture Memory for Indigo2 IMPACT graphics.

### Real-Time Mipmap Generation

If you want your application to use the texture node's ability to do real-time mipmap generation, several issues should be considered. Before a texture is created, the OpenGL minifying filter must be set to a filter that enables mipmapping. You set it by calling the OpenGL function `glTexParameteri(`GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, filter`)` and passing one of the following values for filter:

- GL_NEAREST_MIPMAP_NEAREST

- GL_LINEAR_MIPMAP_NEAREST

- GL_NEAREST_MIPMAP_LINEAR

- GL_LINEAR_MIPMAP_LINEAR

When a texture is created using the OpenGL function `glTexImage2D()`, ten Level-Of-Details (LODs) must be defined (rather than just one for non-mipmap applications). These LODs must be sized so that LOD 0 is 512 horizontally by 256 vertically, and each subsequent LOD (except LOD 9) is half the size, both horizontally and vertically, of the preceding one. LOD 9 must be sized as 1 × 1.

Once the VL texture node control VL_MGV_TEXTURE_MIPMAP_MODE is set to VL_MGV_TEXTURE_MIPMAP_ON, each subsequent call to the OpenGL function `glCopyTexSubImage2DEXT()` generates all ten LODs for the current video field and loads them into TRAM.

To scale the entire video input field in mipmap mode, the zoom and aspect controls must be set correctly. Normally, you set the VL_ZOOM control to 1. For the VL_MGV_HASPECT control, the numerator would be set to 512 and the denominator set to the width of the active video field in pixels. For the VL_MGV_VASPECT control, the numerator would be set to 256 and the denominator set to the larger of 256 and the height of the active video field in lines.

### Timing Issues

For real-time video texturing on an Indigo2 IMPACT workstation equipped with IMPACT graphics and Indigo2 IMPACT Video, these two subsystems must be frame locked. If they are not, you can still perform video texturing, but real-time performance cannot be guaranteed.

To frame-lock the graphics and video hardware, use `setmon`. Before running a video texture application, enter one of these commands:

- NTSC/525-line systems: `/usr/gfx/setmon -Fe 1280x1024_59`

- PAL/625-line systems: `/usr/gfx/setmon -Fe 1280x1024_49`

The texture map interface hardware on the Indigo2 IMPACT Video optional daughter board contains a field buffer that introduces a one-field delay between the input source and the texture memory on the IMPACT graphics board. This one-field delay is hidden by the fact that whenever you call the `glCopyTexSubImage2DEXT()` function to capture a video field, the actual capture does not occur until the beginning of the next video field. This delay occurs regardless of whether the call to `glCopyTexSubImage2DEXT()` occurs during the video vertical blanking interval or during the active video interval.

For example, the `/usr/share/src/dmedia/video/impact/vidtotex.c` program is a simple application of video texturing. It captures each video input field into TRAM, and draws a flat polygon in an X window with it. In this example, the following actions occur:

1. The call to `glCopyTexSubImage2DEXT()` occurs when video field n is being sent into the texture-map interface input select hardware (see Figure 9-1).

2. When the next video field, n + 1, is being sent to the texture map interface hardware, field number n is being transferred out of the field buffer and captured into TRAM on the graphics board, as diagrammed in Figure 9-2.

   Figure 9-2. Video Delay Through Texture-Map Interface Field Buffer

3. During the video field time for field n + 2, field n is being transferred out of TRAM and into the graphics framebuffer, as diagrammed in Figure 9-3.

   Figure 9-3. Video Delay Through Graphics TRAM

4. During the video field time for field n + 3, field n is displayed on the screen, as diagrammed in Figure 9-4.

   Figure 9-4. Video Delay Through Graphics Framebuffer

Thus, for this application, the call to `glCopyTexSubImage2DEXT()` results in the capture of the video field that is entering the texture map interface input select hardware at the time of the call. This field is displayed on the screen three video fields later.

## Controls for Video Texture Mapping

This section describes the controls available on the VL texture drain node. Changing these controls is unrestricted; for example, they can be changed before or after `vlBeginTransfer()` has been called. However, they must match those that are set up through calls to OpenGL; otherwise, the call to `glCopyTexSubImage2DEXT()` returns an error.

### VL_CAP_TYPE

VL_CAP_TYPE specifies the type of field to capture even fields, odd fields, all fields, and non-interleaved frames. Specifying even or odd fields causes each call to OpenGL to initiate a capture to block until the correct field type is present.

If you specify the non-interleaved frame capture type, use the field dominance control (see VL_MGV_DOMINANCE_FIELD, below) to determine the field of the frame to capture first.

### VL_OFFSET

VL_OFFSET specifies the upper left corner of a subregion of the active video region used to generate the texture. The x value is the horizontal offset from the left-hand edge in pixels, and the y value is the vertical offset from the top edge in lines. Along with VL_SIZE, this control can be used to extract any subregion of the input video field.

VL_OFFSET operates on the unzoomed (undecimated) image; it does not change if the zoom factor is changed.

The minimum horizontal offset is 0. The minimum vertical offset depends on the video standard:

- NTSC/525-line: -13

- PAL/625-line: -18 if VL_CAP_TYPE is VL_CAPTURE_ODD_FIELDS; otherwise -19

The maximum horizontal and vertical offsets are restricted: unzoomed size plus offset must not extend past the end of the active video region. The horizontal offset must be an even number.

### VL_PACKING

VL_PACKING specifies the current packing format. For single-link transfers, this control can be set to either VL_PACKING_RGB_8 or VL_PACKING_RGBA_8. For dual-link transfers, this control is always set to VL_PACKING_RGBA_8.

### VL_RATE

VL_RATE is a read-only control that specifies the number of textures per second to capture into TRAM. Since each capture is initiated by a call to OpenGL, you should make at least the same number of calls per second as is specified by this control.

The VLTransferComplete event is generated at the end of a capture into TRAM. The VLSequenceLost event is generated each time a capture into TRAM is missed. However, these events are reported only if they are enabled via the `vlSelectEvents()` routine.

### VL_SIZE

VL_SIZE specifies the amount of the image to send to texture RAM; that is, how much clipping takes place. The x value is interpreted as the subregion width in pixels, and the y value is interpreted as the subregion height in lines.

This control operates on the decimated image. For example, when the image is decimated to half size using VL_ZOOM, the limits of the VL_SIZE control change by a factor of 2.

Along with the VL_OFFSET control, VL_SIZE can be used to extract any subregion of the input video field. Controlling size is useful for cropping bad data at the edges of the video region.

The minimum width is 4 pixels; the minimum height is 3 lines. The maximum width is equal to the active line length multiplied by the product of the zoom and the horizontal aspect values. The maximum height is equal to the total field height minus 4, multiplied by the product of the zoom and the vertical aspect values. The width must be an even number.

    maxwidth = active_line_length x (zoom x horiz_aspect)
    maxheight = (total_field_height - 4) x (zoom x vert_aspect)

### VL_ZOOM

VL_ZOOM specifies the amount of scaling applied to the video subregion used as a texture source. Because only decimation is allowed on the texture node, the zoom value is always less than or equal to 1. For different zoom factors along the horizontal and vertical axes, use VL_MGV_HASPECT and VL_MGV_VASPECT controls. The zoom factor along each axis is equal to the product of the VL_ZOOM value and the aspect value for that axis.

Figure 9-5. Zoom, Size, and Offset for Video Texture Mapping

### VL_MGV_DOMINANCE_FIELD

VL_MGV_DOMINANCE_FIELD specifies the field dominance when VL_CAP_TYPE is set to non-interleaved frames. In that case, VL_MGV_DOMINANCE_FIELD specifies the field of the frame to be captured into TRAM first.

### VL_MGV_HASPECT

VL_MGV_HASPECT specifies the scaling along the horizontal axis that is applied to the video subregion used as a texture source. The overall scale factor horizontally is the product of this value and the VL_ZOOM value. This control enables you to scale the horizontal axis independently of the vertical axis, which is useful when mipmapping is enabled.

To scale the entire video input field horizontally when mipmapping is on, set the zoom to 1, and then set the numerator for VL_MGV_HASPECT to 512 and the denominator to the width of the video field in pixels.

### VL_MGV_VASPECT

VL_MGV_VASPECT specifies the scaling along the vertical axis that is applied to the video subregion used as a texture source. The overall scale factor vertically is the product of this value and the VL_ZOOM value. This control enables you to scale the vertical axis independently of the horizontal axis, which is useful when mipmapping is enabled.

To scale the entire video input field vertically when mipmapping is on, set the zoom to 1, and then set the numerator for VL_MGV_VASPECT to 256 and the denominator to the larger of 256 and the height of the video field in lines.

### VL_MGV_TEXTURE_ROUND_MODE

VL_MGV_TEXTURE_ROUND_MODE specifies the type of rounding used to convert from 10-bit to 8-bit input. You can either round

- normally: VL_MGV_TEXTURE_ROUND_8BIT

- with a pseudonumber generator: VL_MGV_TEXTURE_ROUND_RNG

- with a pseudonumber generator that resets each field: VL_MGV_TEXTURE_ROUND_RNGFRM

### VL_MGV_TEXTURE_MIPMAP_MODE

The VL_MGV_TEXTURE_MIPMAP_MODE control turns mipmap mode off or on. When mipmap mode is on, the texture node generates mipmaps sized as follows:

- 512 × 256

- 256 × 128

- 128 × 64

- 64 × 32

- 32 × 16

- 16 × 8

- 8 × 4

- 4 × 2

- 2 × 1

- 1 × 1

### VL_MGV_TEXTURE_INPUT_LINK

VL_MGV_TEXTURE_INPUT_LINK specifies the input link, A or B, that sends pixel data to TRAM. This control is available only in single-link mode.

### VL_MGV_TEXTURE_AUTOSWAP

VL_MGV_TEXTURE_AUTOSWAP turns autoswap mode off or on. When autoswap mode is on, the input link toggles automatically after each capture into TRAM.

If VL_CAP_TYPE is set to non-interleaved frame, toggling occurs after frame captures. VL_MGV_TEXTURE_AUTOSWAP is useful for applications that require two live video textures simultaneously, and is available only in single-link mode.

## OpenGL Functions for Video Texture Mapping

This section explains how to use OpenGL functions for video texture mapping. For complete information on these functions, see the OpenGL Programming Guide.

- `glGenTexturesEXT`(GLsizei count, GLuint *texnames)

  This function generates texture names, which are unsigned integers, and puts count texture names in texnames. The generated texture names have no dimensionality, but assume that of the texture target to which they are first bound (see `glBindTexureEXT()`, below).

  For applications that use more than one video texture at a time (for example, double-buffering the TRAM loads), use this function to get the names of textures so that they can later be passed to `glBindTexturesEXT()` in order to identify them.

- `glBindTextureEXT`(GLenum target, GLuint texname)

  This function makes it possible to use named textures besides the usual OpenGL texture targets. While a texture is bound, GL operations on the target to which it is bound affect the bound texture. For video texturing applications, target should always be GL_TEXTURE_2D.

- `glHint`(GLenum target, GLenum mode)

  You can control TRAM double buffering with hints (like other aspects of OpenGL behavior). To enable double buffering, call this function with target set to GL_TEXTURE_MULTI_BUFFER_HINT_SGIX and mode set to GL_FASTEST. To disable double buffering, use the same target, but set mode to GL_NICEST.

- `glEnable`(GLenum capability)

  Use this function to enable texturing. Setting capability to GL_TEXTURE_2D enables two-dimensional texturing.

- `glXCreateGLXVideoSourceSGIX`(Display *dpy, int screen, VLServer svr, VLPath path, int nodeClass, VLNode node)

  This function creates a GLX handle for a video input stream that can be used as the read parameter on a call to `glXMakeCurrentReadSGI()`. Thereafter, the GL transfers video data into texture memory when `glCopyTexSubImage2DEXT()` is called.

  If any control that affects the transfer of video data is changed on the video transfer path for which a particular GLX video source was created, destroy the GLX video source and create a new one. Otherwise, the data read from the source will be undefined. The configuration of a GLX video source is static, and is fixed when the GLX video source is created.

- `glXMakeCurrentReadSGI`(Display *dpy, GLXDrawable draw, GLXDrawable read, GLXContext gc)

  This function attaches a GLX context to separate read and write drawables. The handle returned by `glXCreateGLXVideoSourceSGIX()` can be passed as the read parameter. Thereafter, the GL transfers video data into texture memory when `glCopyTexSubImage2DEXT()` is called.

- `glTexParameteri`(GLenum target, GLenum pname, GLint param)

  This function assigns the value in param to the texture parameter specified by pname for the target given by target. To enable or disable mipmapping, set target to GL_TEXTURE_2D and pname to GL_TEXTURE_MIN_FILTER. To turn mipmapping off, set param to either GL_NEAREST or GL_LINEAR. To turn mipmapping on, set param to one of the following: GL_NEAREST_MIPMAP_NEAREST, GL_NEAREST_MIPMAP_LINEAR, GL_LINEAR_MIPMAP_NEAREST, or GL_LINEAR_MIPMAP_LINEAR. See the OpenGL Programming Guide for a description of these parameters.

- `glTexImage2D`(GLenum target, GLint level, GLint components, GLsizei width, GLsizei height, GLint border, GLenum format, GLenum type, const GLvoid *pixels)

  This function defines a texture image. The arguments describe the parameters of the texture image, such as height, width, level-of-detail number, and the internal resolution and format used to store the image.

  The height and width must be powers of two. When mipmapping is not used, only level-of-detail number 0 need be defined; otherwise all ten LODs must be defined. For video textures, pixels can be set to NULL, because the texture data itself is loaded through calls to `glCopyTexSubImage2DEXT()`.

- `glCopyTexSubImage2DEXT`(GLenum target, GLint level, GLint xoffset, GLint yoffset, GLint x, GLint y, GLsizei width, GLsizei height)

  This function replaces a rectangular portion of a two-dimensional texture image with pixels from the current GL_READ_BUFFER. The screen-aligned pixel rectangle with lower left corner at (x,y) having width width and height height replaces the portion of the texture array with x indices xoffset through xoffset + width - 1, inclusive, and y indices yoffset through yoffset + height - 1, inclusive, at the mipmap level specified by level.

  However, for video texture loads, to load all 10 mipmap levels, enable mipmapping using `glTexParameteri()` and call this function with level set to 0. For video textures, this setting initiates a load into texture memory from the video port.

  If any OpenGL texture parameters do not match the corresponding parameters set up through the Video Library, no load occurs and the GL error flag (see `glGetError()`), is set accordingly.

- ```glGetError()`

  This function returns the value of the error flag. Each detectable error is assigned a numeric code and a symbolic constant. When an error occurs, the error flag is set to the appropriate error code value. No other errors are recorded until `glGetError()` is called, the error code is returned, and the flag is reset to GL_NO_ERROR.

  For video texture loads, a mismatch between the OpenGL parameters and the Video Library parameters generates an error that can be detected with this function.

## Example Program: vidtotex.c

The example program `/usr/share/src/dmedia/video/impact/vidtotex.c` continuously captures a stream of live video from a video source and repeatedly loads it into an OpenGL texture map. Those texture maps are then used to texture a polygon in an X window on the IMPACT graphics display.

The video source can be selected via a command-line option, or via the default input device control on `vcp`. The output of `vlinfo` shows valid video source numbers and their mappings.

The program `vidtotex.c` must be run on an Indigo2 IMPACT Graphics system with a TRAM option card installed (Texture Upgrade to 4 MB of Texture Memory for Indigo2 IMPACT graphics).

[3] For a more detailed explanation of the OpenGL routines mentioned in this chapter, see the OpenGL Programming Guide.

Prev | Table of Contents | Next
Chapter 8. Using Optional Color-Space Conversion |  | Appendix A. Video Basics