General Structure


Figure 1
Note: The sizes of the boxes convey no information.
Figure 1 shows a schematic diagram of the software architecture for the Mac Video Library. Dependency of each software component is indicated by "gravity." If a box rests directly on top of another box, then the software component represented by that top box uses part of (makes direct calls to) the software component in the bottom box.

The figure shows that the Mac Video Library (MVL) makes calls directly to D. Pelli's VideoToolbox routines (a minimal set collected together into a library), to the Mac OS video library calls, and also depends on a third library: the Image Manipulation Library (IML). The IML routines allow you to create and manipulate the 2D images and 3D image sequences that you can display with the Mac Video Library.

Note: You are not prevented in any way from using other functions and code from the VideoToolbox, the MacOS, or elsewhere.

Convention: C++ class names in my libraries are (with a few exceptions) prefixed with a three letter acronym taken from the name of the library. Hence we have objects like iml_uniform_rv, iml_ubyte_image, and mvl_mac_clock at our disposal.
 

When using the mvl_video_manager object to display static images or image sequences (movies) you will generally follow the following steps.
  • Allocate a mvl_video_manager object, passing any needed configuration parameters in its constructor.
  • Call one or more initialization member functions of the mvl_video_manager to tell it
  • Use IML objects to create one or more images and/or image sequences.
  • Ask the mvl_video_manager to display these in various ways.

  • Video display calibration

    The mvl_video_manager cannot for itself determine several important facts. These include whether you are using an (ISR) attenuator, what the luminance of the display is, what form of linearization is required, which display you are using, and whether you are using a special video card: the Radius Thunder 30. The mvl_video_manager provides a set of member functions that allow the user to specify these important pieces of information
     
    NOTE: Default values are assigned in the class constructor. These values should be reasonable, but probably not entirely correct. The defaults are: no attenuator, gamma of 2.0, min and max luminance of 0 and 100 candela per square meter, and MVL-style linearization. There is NO DEFAULT for the display to be used: you must specify this for the Video Manager.
     

    Linearization

    Our task in linearization is to produce a sequence of DAC values (perhaps RGB, or reg/green/blue DAC-value triplets) that produce a linear change of perceived luminance on the target display with linear changes of pixel value. That is, we want to to have

            Perceived_Luminance(Pixel_Value) = MinLuminance + Konstant*Pixel_Value

    The relationship of the function that maps a video DAC value to a screen luminance in cathode ray tubes(CRT's)

            L = f(DAC value)

    is generally nonlinear. This is the function normally measured using a photometer. In practice we use the inverse of f(), call it g(), to compute the DAC value required for the requested luminance. The linearization method provided by this package g() takes the form

            DAC value = g(LF)

    where LF is a fraction of the luminance range. If a DAC value of 0 produces LMin, and a maximum dac value (that is, (2**DacBits-1), where ** denotes exponentiation) produces LMax, then LF = (L-LMin)/(LMax-LMin).

    Historically this process (that is, use of g()) has been galled gamma correction because f() was often taken to be of the form alpha+beta*(L**gamma). This is not really adequate for good linearization of any monitor, so we adopt a form for g() of:

            DAC value = exp(p)

    where exp(p) is the Natural Exponential function, p is a polynomial in ln(LF), and ln() is the Natural Logarithm. The nth order polynomial p has parameters {a0, a1, ... an}. A first order polynomial produces a two parameter a0*(L**(1/a1)). This is equivalent to the historical gamma function: Beta*L**Gamma where Beta=a0 and Gamma=(1/a1). Since a first order polynomial is not generally adequate for all hardware we allow higher order polynomials in our calibration.

     

    Attenuator Characteristics

    The library assumes that, if you have an attenuator, it has the same construction as that of those produced by the Institute for Sensory Perception (ISR). That attenuator scales the red gun to provide around about 32 steps of luminance red voltage for each voltage step of the blue gun. By summing the two together we get higher luminance resolution for the display. If you are using an (ISR) attenuator, you need to specify the number of steps the red gun takes for each step of the blue gun. Generally, the resistive network (e.g. Video Attenuator) produces a (linear) weighted sum of the voltages from the red, blue and green guns and the result is output onto the green channel:

            Vgreen_out = Ar*Vred + Ag*Vgreen + Ab*Vblue

    where Ar, Ag, and Ab are constants that satisfy

            Ar + Ag + Ab = 1.0

    and where (at least, for the Video Attenuator)

            Ab > Ag > Ar

    For the ISR Video Attenuator, Ar > Ab/256. Assuming this to be true for any attanuator used with this software, we use the red gun (alone) to interpolate between luminances produced by integral values of the blue gun. To measure this, the calibration program included with this software finds the red gun dac value that makes

            BlueDacValue + RedDacValue == BlueDacValue+Delta

    true for some Delta. The number of red gun steps per blue step is all we need to know, assuming that the gains for each of the red, green and blue guns are independent. (This is unlikely, but the  assumption works well enough.) This 'step-count' is used in the linearization process.

     

    Mac OS and Video ModeInformation

    Most of this information is specified once you have indicated which video display is being used. The information is gleaned from a set of MacOS function calls, and some of it is just a reflection of user supplied values.
     
    Note: Specification of the video display to be used should be performed before specification of gamma if you are using the MVL calibration. This is because you probably want the gamma curve to max out at MaxDac, a variable whose value you might not know until you have selected a display. For example, Radius Thunder 30 cards have 10 bit DAC's, while standard (built-in) video DACs are 8 bits.
     

    The mvl_video_manager Object

    This sequence of actions described above is done with a set of classes detailed in the header files, consisting primarily of
    The mvl_video_manager does the lion's share of the work, while the mvl_clut, mvl_screen_position and iml_sequence classes are used in conjunction with the the Video Manager to specify various kinds of information. Figure 2 shows a schematic of the functions provided by the mvl_video_manager class.
     


    Figure 2
    Note: The sizes of the boxes convey no information.

    Displaying images and movies

    This, of course, is the primary purpose for building this package. The process is as straightforward as I could make it. The series of actions consists of
  • allocate either a signed or unsigned byte image: iml_byte_image or iml_ubyte_image (2 or 3 dimensional)
  • fill it with whatever image sequence you like
  • then do one of two things:
  • Figure 3 shows the software architecture of your program as it relates to the MVL and IML libraries, together with how those two packages work with system memory and application heap. As the figure indicates, direct image display moves the image straight to the on-screen graphics window without making a copy in off-screen graphics memory. Direct image display (copying from application heap) cannot provide synchronization with vertical refresh. This can result in a "scroll-on" effect when large images are copied to the screen this way.
     
    Blitting the image from off-screen graphics memory allows faster blitting. More importantly, blitting from off-screen graphics memory includes synchronization to the video vertical refresh, allowing the image frame rate to be precisely controlled.
     
    NOTE: If you don't need to show movies, you don't necessarily need to store your image sequences in off-screen graphics memory.

    Figure 3
    Figure 3
    Note: The sizes of the boxes convey no information.

    A code fragment example of how to display an image directly is:
    // ALLOCATE AN IMAGE X Y IN APPLICATION HEAP 
    iml_ubyte_image MyImage(128,128); 

    // FILL THE IMAGE WITH SOMETHING 
    //.... 
    // DECIDE WHERE TO PUT THE IMAGE 
    mvl_screen_position Position(50,50); 

    // SHOW THE IMAGE BY COPYING IT FROM HEAP 
    // TO THE ON-SCREEN GRAPHICS WINDOW 
    MacVideo.display_image(MyImage, Position);

    A code fragment example of how to display an image sequence as a movie:
     
    // ALLOCATE AN IMAGE    X    Y    T 
    // IN APPLICATION HEAP 
    iml_ubyte_image MyImage(64, 64, 32); 

    // FILL THE IMAGE WITH SOMETHING 
    // . . .. 

    // STORE THE IMAGE SEQUENCE IN 
    // OFF-SCREEN GRAPHICS MEMORY 
    IML_SINT ImageID = MacVideo.store_image_sequence(MyImage); 

    // MAKE AN INDEX SEQUENCE; ONE FOR EACH FRAME 
    iml_imageframe_sequence BlitSeq(32); 
    BlitSeq.generate_linear_sequence(0,1); 

    // DECIDE WHERE TO PUT THE IMAGE 
    mvl_screen_position Position(50,50); 

     // MAKE A BLIT SEQUENCE INSIDE THE VIDEO MANAGER 
    IML_SINT BlitID = MacVideo.create_blit_sequence(ImageID, Position, BlitSeq); 

    // SHOW THE IMAGE BY SEQUENTIALLY 
    // BLITTING FRAMES FROM OFF-SCREEN 
    // GRAPHICS MEMORY TO THE ON-SCREEN GRAPHICS WINDOW 
    MacVideo.play_blit_sequence(BlitID);

    NOTE: There are other variations on this. Take a look at the detailed documentation.
     


    CLUT Manipulation and Loading

    If we are to do pyschophysics by displaying images or movies we need to control the appearance of the pixels on the display. The mvl_video_manager class provides a number of methods for manipulating the Color Lookup Table (CLUT) once the proper calibration has been initiated (see above).
     
    NOTE: This works with either form of calibration: D.G. Pelli's or the simple gamma correction provided here.
     
    The basic requirement, generally, is to produce a sequence of CLUT entries which translate sequential pixel values (in the range 0 to 255) into a linearly spaced range of screen luminance values. Unless the monitor has a gamma of 1.0 (unlikely), the spacing of DAC values required for linear steps of luminance is not a constant. This means that, although we might have a CLUT of 256 entries, the actual luminance resolution will be less than 8 bits. The standard method for using the calibration information to produce a linearizing set of CLUT entries is the linearize_clut() member function. In the form
     
    float MeanL = 0.5; 
    float MaxC = 1.0; 
    MacVideo.linearize_clut(MeanL, MaxC);
    This fills the default CLUT segment with a linearized sequence. The default CLUT segment is entry 1 to entry (MacVideo.CLUTSize-2), inclusive. Apple defaults to using entry 0 as maximum luminance and entry (MacVideo.CLUTSize-1) as minimum luminance for text displays in the console window. The first and last entries are are preserved by default because then you only need one monitor for showing images and reading/writing/typing text in the console window.
     
    If you use two monitors, one for display and one for the console, then you can change the default to use the entire CLUT with:
     
    MacVideo.set_default_clut_segment(0, MacVideo.CLUTSize);
    Alternatively, you can use a smaller CLUT segment, or even multiple linearized CLUT segments using:
     
    // TWO LINEARIZED SEGMENTS WITH 20% AND 40% CONTRAST RANGES 

    float MeanL = 0.5, MaxC = 0.2; 
    unsigned short Start = 1, Length = 100; 
    MacVideo.linearize_clut(MeanL, MaxC, Start, Length); 
    MacVideo.linearize_clut(MeanL, 2*MaxC, Start+Length, Length);

    If this is not enough control for you (vision scientists like to be in control, you know) then you can always grab a copy of the CLUT, make whatever changes you like, then load it back in. This looks like:
     
    // GET A COPY OF THE CLUT CURRENTLY IN THE HARDWARE 
    mvl_clut *CLUTCopy = MacVideo.get_clut_copy(); 

    // SET CLUT ENTRY 127 TO THE COLOR I WANT 
    short Entry1 = 127, Red = 100, Green = 23, Blue = 250; 
    CLUTCopy->set_clut_entry(Entry1, Red, Gree, Blue) 

    // SET THE RED GUN IN CLUT ENTRY 130 TO 200 
    short Entry2 = 130; 
    CLUTCopy->red(Entry2, 200) 

    // LOAD THE CLUT INTO THE HARDWARE 
    // AND MAKE THE DEFAULT CLUT 
    MacVideo.make_default_clut(CLUTCopy); 

    You can access the individual mvl_ColorSpec structures directly using array indexing (well, using array syntax with operator[], for you C++ folks). However the routines provided know how to correctly left-justify the DAC values in the mvl_rgb data structure, given the DAC width (in bits). If you don't do that correctly, then you won't get the color/luminance that you really want on the screen. :-)