Name NV_vdpau_interop Name Strings GL_NV_vdpau_interop Contributors Stephen Warren, NVIDIA James Jones, NVIDIA Contact Stephen Warren, NVIDIA (swarren 'at' nvidia.com) Status Complete. Version 8 (24 Feb 2010) Number 1 Dependencies This extension is written against the OpenGL 3.2 Specification but can apply to OpenGL 1.1 and up. OpenGL 1.1 is required. GL_EXT_framebuffer_object affects the definition of this extension. GL_ARB_texture_rectangle affects the definition of this extension. GL_ARB_texture_non_power_of_two affects the definition of this extension. Overview This extension allows VDPAU video and output surfaces to be used for texturing and rendering. This allows the GL to process and display the content of video streams decoded using VDPAU. Alternatively, the GL may modify VDPAU surfaces in-place, and VDPAU maythen process and/or display those surfaces itself. This allows the GL to be used to combine application user-interface elements with decoded video, implement custom video-processing algorithms, etc. IP Status There are no known IP issues. New Procedures and Functions void VDPAUInitNV (const void *vdpDevice, const void *getProcAddress); void VDPAUFiniNV (void); vdpauSurfaceNV VDPAURegisterVideoSurfaceNV (const void *vdpSurface, enum target, sizei numTextureNames, const uint *textureNames); vdpauSurfaceNV VDPAURegisterOutputSurfaceNV (const void *vdpSurface, enum target, sizei numTextureNames, const uint *textureNames); void VDPAUIsSurfaceNV (vdpauSurfaceNV surface); void VDPAUUnregisterSurfaceNV (vdpauSurfaceNV surface); void VDPAUGetSurfaceivNV (vdpauSurfaceNV surface, enum pname, sizei bufSize, sizei *length, int *values); void VDPAUSurfaceAccessNV (vdpauSurfaceNV surface, enum access); void VDPAUMapSurfacesNV (sizei numSurfaces, const vdpauSurfaceNV *surfaces); void VDPAUUnmapSurfacesNV (sizei numSurface, const vdpauSurfaceNV *surfaces); New Tokens Accepted as the parameter of VDPAUGetSurfaceivNV: SURFACE_STATE_NV 0x86EB Returned in for VDPAUGetSurfaceivNV SURFACE_STATE_NV: SURFACE_REGISTERED_NV 0x86FD SURFACE_MAPPED_NV 0x8700 Accepted as the parameter of VDPAUSurfaceAccessNV: WRITE_DISCARD_NV 0x88BE Additions to Chapter 2 of the OpenGL 3.2 (unabridged) Specification (OpenGL Operation) Add to Table 2.2, GL data types: GL Type Minimum Description Bit Width ------- --------- ---------------------------------------------- vdpauSurfaceNV VDPAU surface object handle (see section 3.8.3) Additions to Chapter 3 of the OpenGL 3.2 (unabridged) Specification (Rasterization) Insert a new section following "Alternate Texture Image Specification Commands" (Section 3.8.2). Renumber existing section 3.8.3 "Compressed Texture Images" and all following 3.8.* sections. 3.8.3, Alternate Texture Image Specification Using VDPAU Surfaces ----------------------------------------------------------------- Texture images may also be specified using image data taken directly from the framebuffer, in the form of VDPAU surfaces. VDPAU surfaces are logically ascribed an interop state. States and transitions are: VDPAUUnegisterSurfaceNV/VDPAUFiniNV /----------------------------------------\ | | v VDPAURegister*SurfaceNV | Unknown/unregistered ----------------------> Registered ^ | ^ | VDPAUMapSurfacesNV | | VDPAUUnmapSurfacesNV | v | \------------------------------------- Mapped VDPAUFiniNV In order to use such "interop" functionality, the command void VDPAUInitNV (const void *vdpDevice, const void *getProcAddress); must be used to inform the GL which VDPAU device to interact with. must be an extant VdpDevice handle previously created using VDPAU. must be the VdpGetProcAddress generated during VdpDevice creation. When interop functionality is no longer required, the GL may be informed using the command void VDPAUFiniNV (void); This operation will succeed even if surfaces exist in the mapped or registered states; such surfaces will internally be unmapped and unregistered before interop state is destroyed. The command vdpauSurfaceNV VDPAURegisterVideoSurfaceNV (const void *vdpSurface, enum target, sizei numTextureNames, const uint *textureNames); defines a set of two-dimensional textures, where the image data may be taken from the VdpVideoSurface . must be one of TEXTURE_2D or TEXTURE_RECTANGLE. determines how many textures are defined. contains the names of the textures that are defined. The surface is transitioned into the registered state. Legal values for are derived from the VdpChromaType of , as defined in table 3.8.3.1. Internal VdpChromaType numTextureNames Index Size Format Content ------------- --------------- ----- ---- -------- ------------------- VDP_CHROMA_TYPE_420 4 0 w x h/2 R8 Top-field luma 1 w x h/2 R8 Bottom-field luma 2 w/2 x h/4 R8G8 Top-field chroma 3 w/2 x h/4 R8G8 Bottom-field chroma VDP_CHROMA_TYPE_422 4 0 w x h/2 R8 Top-field luma 1 w x h/2 R8 Bottom-field luma 2 w/2 x h/2 R8G8 Top-field chroma 3 w/2 x h/2 R8G8 Bottom-field chroma Table 3.8.3.1: Supported VdpChromaType values, and derived values of , and texture parameters for each texture. VDPAURegisterVideoSurfaceNV's return value is a handle used by various other commands detailed below. The command vdpauSurfaceNV VDPAURegisterOutputSurfaceNV (const void *vdpSurface, enum target, sizei numTextureNames, const uint *textureNames); defines a set of two-dimensional textures, where the image data may be taken from the VdpOutputSurface vdpSurface. must be one of TEXTURE_2D or TEXTURE_RECTANGLE. determines how many textures are defined. contains the names of the textures that are defined. The surface is transitioned into the registered state. Legal values for are derived from the VdpRGBAFormat of , as defined in table 3.8.3.2. Internal VdpRGBAFormat numTextureNames Index Size Format Content ------------- --------------- ----- ---- -------- -------------- VDP_RGBA_FORMAT_B8G8R8A8 1 0 w x h ARGB8 Entire surface VDP_RGBA_FORMAT_R10G10B10A2 1 0 w x h A2BGR10 Entire surface Table 3.8.3.2: Supported VdpRGBAFormat values, and derived values of , and texture parameters for each texture. In all cases, textures defined by VDPAU surfaces will be y- inverted; applications are expected to use y-inverted texture co- ordinates when using such textures. VDPAURegisterOutputSurfaceNV's return value is a handle to be used as the surface parameter to various other commands detailed below. The command void VDPAUUnregisterSurfaceNV (GLvdpauSurfaceNV surface); may be used to revert the effect of VDPAURegisterVideoSurfaceNV or VDPAURegisterOutputSurfaceNV; namely, any s that were passed into the register function will be returned to their default state. The surface will be transitioned into the unregistered state. This operation will succeed even if the surface is in the mapped state; such surfaces will internally be unmapped before being unregistered. The command void VDPAUSurfaceAccessNV (GLvdpauSurfaceNV surface, enum access); informs the GL whether the application expects only to read from the VDPAU surface using the GL ( set to READ_ONLY), only to write to the VDPAU surface using the GL ( set to WRITE_DISCARD_NV), or both ( set to READ_WRITE). In implementations that need to copy image data during VDPAUMapSurfacesNV or VDPAUUnmapSurfacesNV, this flag may allow the GL to skip some of those copy operations to improve performance. An value of READ_ONLY ensures that any writes to the surface made by VDPAU prior to VDPAUMapSurfacesNV will be seen by the GL when reading the surface after VDPAUMapSurfacesNV. If the GL writes to the surface, results are undefined, possibly including program termination. An value of READ_WRITE ensures that any writes to the surface made by VDPAU prior to VDPAUMapSurfacesNV will be seen by the GL when reading the surface after VDPAUMapSurfacesNV. Equally, any writes to the surface made by the GL prior to VDPAUUnmapSurfacesNV will be seen by VDPAU after VDPAUUnmapSurfacesNV. An value of WRITE_DISCARD_NV ensures that any writes to the surface made by the GL prior to VDPAUUnmapSurfacesNV will be seen by VDPAU after VDPAUUnmapSurfacesNV. If the GL reads from the surface prior to writing those particular pixels itself, the graphical results are undefined, but should not include program termination. VDPAUSurfaceAccessNV sets the access flag for the VDPAU surface. the GL reads the access flag only during VDPAUMapSurfacesNV. VDPAUUnmapSurfacesNV uses the value of the access flag that was set when the previous VDPAUMapSurfacesNV was called. VDPAUSurfaceAccessNV itself has no effect on the image data of any surface, and in particular does not initiate any copy operations. Calling VDPAUSurfaceAccessNV when the surface is in a mapped state is illegal, and will generate an error. The command void VDPAUMapSurfacesNV (sizei numSurfaces, vdpauSurfaceNV *surfaces); moves a set of VDPAU surfaces from the registered state into the mapped state. The command void VDPAUUnmapSurfacesNV (sizei numSurface, vdpauSurfaceNV* surfaces); moves a set of VDPAU surfaces from the mapped state into the registered state. Using the GL to texture from, or write to, a surface in any state other than mapped will yield undefined results, potentially including program termination. Using any other API to access a VDPAU surface while that surface is in the mapped state in the GL will yield undefined results, potentially including program termination. Texture image data for VDPAU surfaces in the mapped state is defined as being identical to the VDPAU surface content. While a VDPAU surface is in the mapped or registered states, the textures associated with that surface are immutable. In other words, operations that affect texture state, or attempt to replace the source of the texture image data (e.g. TexImage2D) will return an error. Rendering to the texture is allowed. Additions to Chapter 4 of the OpenGL 3.2 (unabridged) Specification (Per-Fragment Operations and the Frame Buffer) None Additions to Chapter 5 of the OpenGL 3.2 (unabridged) Specification (Special Functions) None Additions to Chapter 6 of the OpenGL 3.2 (unabridged) Specification (State and State Requests) Add a new subsection to "Querying GL state" (section 6.1) describing VDPAU surface object queries. "6.1.X VDPAU Surface Object Queries Properties of VDPAU surface objects may be queried using the command void VDPAUGetSurfaceivNV (vdpauSurfaceNV surface, enum pname, sizei bufSize, sizei *length, int *values); The value or values being queried are returned in the parameters and . On success, VDPAUGetSurfaceivNV replaces up to integers in with the corresponding property values of the object being queried. The actual number of integers replaced is returned in *. If is NULL, no length is returned. If is SURFACE_STATE_NV, a single value representing the state of the VDPAU surface object (SURFACE_REGISTERED_NV, SURFACE_MAPPED_NV) is placed in . If is not the name of a VDPAU surface object, an INVALID_VALUE error is generated. If is not one of the values described above, an INVALID_ENUM error is generated. If an error occurs, nothing will be written to or . The command void VDPAUIsSurfaceNV (vdpauSurfaceNV surface); returns TRUE if is the name of a VDPAU surface object. If is not the name of a VDPAU surface object, or if an error condition occurs, IsSurface returns FALSE (note that zero is not the name of a VDPAU surface object). Additions to the AGL/GLX/WGL Specifications None Additions to the OpenGL Shading Language None GLX Protocol VDPAU implementations currently only support direct-rendering. Consequently, no GLX protocol is currently defined for this extension. Dependencies on GL_ARB_texture_rectangle If GL_ARB_texture_rectangle is not supported, TEXTURE_RECTANGLE may not be used as target for VDPAURegisterVideoSurfaceNV. Dependencies on GL_ARB_texture_non_power_of_two If GL_ARB_texture_non_power_of_two is not supported, only VDPAU surfaces with power-of-two size may be used with target TEXTURE_2D. Errors INVALID_VALUE is generated if the or parameters to VDPAUInitNV is NULL. INVALID_OPERATION is generated if VDPAUInitNV is called on a given GL context, where VDPAUInitNV has been called, and VDPAUFiniNV has not been called since. INVALID_OPERATION is generated by VDPAUInitNV, VDPAURegisterVideoSurfaceNV, or VDPAURegisterOutputSurfaceNV if the VDPAU driver refuses the request for some reason. INVALID_OPERATION is generated if VDPAUFiniNV, VDPAURegisterVideoSurfaceNV, VDPAURegisterOutputSurfaceNV, VDPAUIsSurfaceNV, VDPAUGetSurfaceivNV, VDPAUSurfaceAccessNV, VDPAUMapSurfacesNV, or VDPAUUnmapSurfacesNV are called on a given GL context, where VDPAUInitNV has not been called, or VDPAUFiniNV has been called since. INVALID_ENUM is generated if the parameter of VDPAURegisterVideoSurfaceNV or VDPAURegisterOutputSurfaceNV is not TEXTURE_2D or TEXTURE_RECTANGLE. INVALID_VALUE is generated if the parameter of VDPAURegisterVideoSurfaceNV or VDPAURegisterOutputSurfaceNV does not match the required value; see tables 3.8.3.1 and 3.8.3.2. INVALID_OPERATION is generated if any texture named by an entry within the parameter of VDPAURegisterVideoSurfaceNV or VDPAURegisterOutputSurfaceNV is marked as immutable. INVALID_VALUE is generated if the VDPAU surface named by the parameter of VDPAURegisterVideoSurfaceNV or VDPAUUnregisterOutputSurfaceNV does not have a supported format; see tables 3.8.3.1 and 3.8.3.2. INVALID_VALUE is generated if the parameter of VDPAUIsSurfaceNV, VDPAUGetSurfaceivNV, VDPAUSurfaceAccessNV, VDPAUMapSurfacesNV, VDPAUUnmapSurfacesNV is not the name of a VDPAU surface object. INVALID_VALUE is generated if the parameter of VDPAUUnregisterSurfaceNV is neither zero nor the name of a VDPAU surface object. INVALID_VALUE is generated if the parameter of VDPAUGetSurfaceivNV is too small to return the results via the paramter. INVALID_ENUM is generated if the parameter of VDPAUGetSurfaceivNV is not SURFACE_STATE_NV. INVALID_VALUE is generated if the parameter of VDPAUSurfaceAccessNV is none of READ_ONLY, WRITE_ONLY, nor READ_WRITE. INVALID_OPERATION is generated if the state of the surface(s) named by the parameter of VDPAUSurfaceAccessNV, or named by any entry within the parameter of VDPAUMapSurfacesNV is SURFACE_MAPPED_NV. INVALID_OPERATION is generated if the state of any of the surfaces named by entries within the parameter of VDPAUUnmapSurfacesNV is not SURFACE_MAPPED_NV. New State Table 6.X. VDPAU Surface Objects. Get value Type Get command Initial value Description Section ------------------ ---- ----------- ---------------------------- --------------- ------- SURFACE_STATE_NV Z VDPAUGetSurfaceivNV GL_SURFACE_REGISTERED_NV Surface state 3.8.3 New Implementation State None Issues 1. Should YUV surfaces be exposed as a frame Y and frame UV texture, or two field Y and two field UV textures? RESOLVED: YUV surfaces will be exposed as separate fields Exposing the surface as separate fields allows applications to directly implement de-interlacing algorithms other than weave without having to undo the weaving of fields together. Put another way: Frames: Pros: * Simple applications can directly access the entire frame, without having to manually implement weave de-interlacing. This also applies for progressive content. However, simple applications probably won't want to interop on a VdpVideoSurface, but will rather rely on VDPAU's VdpVideoMixer for post-processing, and instead interop on a VdpOutputSurface. Cons: * In order to implement bob de-interlacing, vertical interpolation between lines of a single field is required. The GL's texturing does not allow interpolation to skip lines in a texture. Consequently, the two fields must be manually separated by the application using a shader and some texture co-ordinate calculation prior to bobbing. This forces a copy operation, which would reduce performance. * In NVIDIA's implementation, VDPAU surfaces are stored as separate fields. Consequently, VDPAUMapSurfacesNV would have to perform a copy operation, which would reduce performance. Fields: Pros: * Advanced de-interlacing and post-processing algorithms get direct access to individual fields. * In NVIDIA's implementation, no copying is required during VDPAUMapSurfacesNV. Cons: * Simple applications that simply want to weave, or for progressive content, must manually combine the fields. However, this operation is relatively simple to code, and sample code will be provided. 2. Should Map/Unmap functions be removed, such that the GL implements those operations transparently at appropriate times? RESOLVED: No, map/unmap functions will be kept. The map/unmap functions provide a convenient place to perform synchronization between VDPAU and the GL command streams, and any format-converting copies that may be required by the GL or VDPAU implementation. Without explicit application-controlled APIs, it would potentially be difficult to trigger those operations at the appropriate time. It would also be less obvious to software developers when such operations were occurring, which may prevent successful investigation of interop performance. 3. Should SurfaceAccess function be removed, and instead an "access" parameter added to the Map function? RESOLVED: No, a separate function will be kept. The value of this flag typically will not change on a per-frame basis, so specifying it for each map call was deemed redundant. For an advanced application that needs to change the flag often, the overhead of using a separate function should not be too significant. Revision History 1. 23 Dec 2009 - Stephen W Initial version 2. 5 Jan 2010 - Stephen W -Renamed VDPAUSetSurfaceAccessNV to VDPAUSurfaceAccessNV thus avoiding the word Set. -Fixed/wrote "Dependencies", "Dependencies on ..." sections. -Initial version of "Additions to Chapter 3". -Fill in answers for issues 1-3. 3. 7 Jan 2010 - Stephen W -Dropped the GL prefix from type names. -Refer to "the GL" not "GL". -Removed XXX re: PROXY_TEXTURE_*; those targets don't seem useful for this extension. -Use <> around function parameter names in text. -Reword description of legal values for . Number, title, and reformat related tables. Rename "format" column to "internal format". -Describe VDPAUSurfaceAccessNV before Map/Ummap for more logical ordering. -Significant text rewrite to more explicitly describe states, state transitions, legal access to surfaces by various APIs during specific states, etc. -Explicitly state that registered/mapped textures are immutable. -Re-used VBO's READ/WRITE/READ_WRITE enums instead of inventing new enums. -Added explicit type GLvdpauSurfaceNV instead of using GLvoid*. -Added VDPAUIsSurface, VDPAUGetSurfaceiv, and associated tokens. -Documented state, state queries, and errors. 4. 13 Jan 2010 - Stephen W - Added token WRITE_DISCARD_NV. Stated semantics of each legal value of VDPAUSurfaceAccessNV's parameter. 5. 13 Jan 2010 - Stephen W - Removed some XXXs. Marked spec as complete. 6. 18 Jan 2010 - Stephen W - Added missing NV suffix to VDPAUIsSurfaceNV, VDPAUGetSurfaceivNV 7. 18 Jan 2010 - Made various function parameters const. 8. 24 Feb 2010 - Fixed typo in table 3.8.3.1; the heights of luma surfaces were incorrectly specified.