® Video Quality Research

Video Home   |    ITS Home

Documents

These documents are arranged in chronological order, starting with the most recent publications.

• Reduced Reference Video Calibration Algorithms, National Telecommunications and Information Administration (NTIA) Technical Report TR-08-433b, November, 2007.

  • Authors: Margaret H. Pinson, Stephen Wolf

  • ABSTRACT: This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network connection between the original (source) and processed (destination) ends. The first RR video calibration algorithm computes temporal registration of the processed video stream with respect to the original video stream (i.e., video delay estimation). The second algorithm jointly calculates spatial scaling and spatial shift. The third algorithm calculates luminance gain level offset of the processed video stream with respect to the original video stream. The fourth algorithm estimates the valid video region of the original or processed video stream (i.e., the portion of the video image that contains actual picture content). All the algorithms utilize only the luminance (Y) image plane of the video signal.

  
  

• Batch Video Quality Metric (BVQM) User’s Manual, National Telecommunications and Information Administration (NTIA) Handbook HB-08-441b, November, 2007.

  • Authors: Margaret H. Pinson, Stephen Wolf

  • ABSTRACT: This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output by a video system under test). BVQM reports video calibration and quality metric results such as: temporal registration, spatial registration, spatial scaling, valid region, gain/level offset, and objective video quality estimates. BVQM operates on original and processed video files only, and has no video capture capability.

    BVQM compares the original video clip to the processed video clip and reports quality estimates on a scale from zero to one. On this scale, zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level (excursions beyond one are possible for extremely impaired video sequences).

  
  
• Video Performance Requirements for Tactical Video Applications, 2007 IEEE Conference on Technologies for Homeland Security, Woburn, MA, May 16-17, 2007.
  
  
  • Authors: Margaret H. Pinson, Stephen Wolf, and Robert B. Stafford
    
    
  • ABSTRACT: The Public Safety Statement of Requirements (PS-SoR) for Communications and Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form that allows the practitioners to effectively use it. PS-SoR Volume I defined functional communication and interoperability requirements. Published in September, 2006, PS-SoR Volume II identifies quantitative performance metrics, including minimum video performance requirements for public safety's tactical video applications. The goal was not to identify what is achievable with current technology but rather, looking towards the future, to investigate the minimum level of performance that first responders need in order to effectively use their video equipment.
    
    On behalf of the SAFECOM Program and the Office of Law Enforcement Standards, the Institute for Telecommunication Sciences (ITS) conducted subjective video quality testing to estimate the level of video quality that first responders find acceptable for tactical video applications. This subjective testing utilized source video content that is typical of public safety operations in structured subjective viewing experiments with 35 first responders. The evaluations from these first responders, in viewing high quality video (original video) and purposefully degraded video (using video compression and transmission equipment), allowed determination of basic quality thresholds for public safety tactical video applications. These perceptual quality thresholds have been translated into technical parameters for use by video equipment designers, manufacturers, and customers. This paper summarizes those findings. Other testing to evaluate requirements for other public safety applications is underway.
  
  
• Application of the NTIA General Video Quality Metric (VQM) to HDTV Quality Monitoring, Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM-07), Scottsdale, Arizona, January 25-26, 2007.
  
  
  • AUTHORS: Stephen Wolf and Margaret H. Pinson
    
    
  • ABSTRACT: This paper summarizes results from an experiment whose goal was to assess whether the NTIA General Video Quality Metric (VQM) is an acceptable objective metric for measuring High Definition TV (HDTV) video quality. The HDTV subjective test that was performed to evaluate the NTIA General VQM contained 60 30-second video clips that were rated using the Single Stimulus Continuous Quality Evaluation (SSCQE) method. The 60 clips included twelve 1080i HDTV originals and 48 processed versions of these originals from 16 different video systems. The video systems included 5 different HDTV codecs running at bit rates from 2 to 19 Mbps and broadcast transmission errors (i.e., RF transmission with poor signal-to-noise-ratio). Excellent objective-to-subjective correlation results for this experiment demonstrate the potential application of the NTIA General VQM to HDTV quality monitoring.
  
  
• Batch Video Quality Metric (BVQM) User’s Manual, National Telecommunications and Information Administration (NTIA) Handbook HB-06-441a, December, 2006.
  
  
  • AUTHORS: Mark A. McFarland, Margaret H. Pinson, Stephen Wolf
    
    
  • ABSTRACT: This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® operating system. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output by a video system under test). BVQM reports video calibration and quality metric results such as: temporal registration, spatial registration, spatial scaling, valid region, gain/level offset, and objective video quality estimates. BVQM operates on original and processed video files only, and has no video capture capability.
    
    BVQM compares the original video clip to the processed video clip and reports quality estimates on a scale from zero to one. On this scale, zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level (excursions beyond one are possible for extremely impaired video sequences).
    
    
  • KEYWORDS: automatic measurements; batch video clip processing; digital video; metrics; objective video quality performance; video calibration; video quality
  
  
• In Service Video Quality Metric (IVQM) User's Manual (a), National Telecommunications and Information Administration (NTIA) Handbook HB-06-434a, July, 2006.
  
  
  • AUTHORS: Margaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicating through an IP connection. IVQM performs image acquisition, temporal registration, other video calibration (spatial registration, spatial scaling, valid region, and gain/level offset), and video quality estimation.
    
    IVQM compares the source video sequence to the destination video sequence (i.e., as output by the video system under test). Each program alternates between video capture and video analysis. Every source/destination video sequence pair is processed through three main steps. First, the sequences are buffered onto a hard drive. Second, the sequences are temporally registered. Third, the video quality of the destination video sequence is estimated. Quality estimates are reported on a scale of zero to one, where zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level. Some video sequences may also be used to estimate other calibration values (spatial registration, spatial scaling, valid region estimation, and gain/level offset). The user has control over how often these other calibration values are calculated.
    
    
  • KEYWORDS: automatic measurements; digital video; end-to-end; in-service; metrics; objective video quality performance; video calibration; video quality
  
  
• Reduced Reference Video Calibration Algorithms, National Telecommunications and Information Administration (NTIA) Technical Report TR-06-433a, July, 2006.
  
  
  • AUTHORS: Margaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network connection between the original (source) and processed (destination) ends. The first RR video calibration algorithm computes temporal registration of the processed video stream with respect to the original video stream (i.e., video delay estimation). The second algorithm jointly calculates spatial scaling and spatial shift. The third algorithm calculates luminance gain level offset of the processed video stream with respect to the original video stream. The fourth algorithm estimates the valid video region of the original or processed video stream (i.e., the portion of the video image that contains actual picture content). All the algorithms utilize only the luminance (Y) image plane of the video signal.
    
    
  • KEYWORDS: Calibration; delay; gain; offset; spatial scaling; spatial shift; temporal shift; video
  
  
• In Service Video Quality Metric (IVQM) User's Manual, National Telecommunications and Information Administration (NTIA) Handbook HB-06-434, December, 2005.
  
  
  • AUTHORS: Margaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: The purpose of this handbook is to provide a user's manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicating through an IP connection. IVQM performs image acquisition, temporal registration, other video calibration (spatial registration, spatial scaling, valid region, and gain/level offset), and video quality estimation. IVQM compares the source video sequence to the destination video sequence (i.e., as output by the video system under test). Each program alternates between video capture and video analysis. Every source/destination video sequence pair is processed through three main steps. First, the sequences are buffered onto a hard drive. Second, the sequences are temporally registered. Third, the video quality of the destination video sequence is estimated. Quality estimates are reported on a scale of zero to one, where zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level. Some video sequences may also be used to estimate other calibration values (spatial registration, spatial scaling, valid region estimation, and gain/level offset). The user has control over how often these other calibration values are calculated.
    
    
  • KEYWORDS: automatic measurements; digital video; end-to-end; in-service; metrics; objective video quality performance; video calibration; video quality
  
  
• Reduced Reference Video Calibration Algorithms, National Telecommunications and Information Administration (NTIA) Technical Report TR-06-433, October, 2005.
  
  
  • AUTHORS: Margaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network connection between the original (source) and processed (destination) ends. The first RR video calibration algorithm computes temporal registration of the processed video stream with respect to the original video stream (i.e., video delay estimation). The second algorithm jointly calculates spatial scaling and spatial shift. The third algorithm calculates luminance gain level offset of the processed video stream with respect to the original video stream. The fourth algorithm estimates the valid video region of the original or processed video stream (i.e., the portion of the video image that contains actual picture content). All the algorithms utilize only the luminance (Y) image plane of the video signal.
    
    
  • KEYWORDS: Calibration; delay; gain; offset; spatial scaling; spatial shift; temporal shift; video
  
  
• Low Bandwidth Reduced Reference Video Quality Monitoring System, First International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Scottsdale, Arizona, January 23-25, 2005.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: This paper presents a new reduced reference (RR) video quality monitoring system that utilizes less than 10 kbits/s of reference information from the source video stream. This new video quality monitoring system utilizes feature extraction techniques similar to those found in the NTIA General Video Quality Model (VQM) that was recently standardized by the American National Standards Institute (ANSI) and the International Telecommunication Union (ITU). Objective to subjective correlation results are presented for 18 subjectively rated data sets that include more than 2500 video clips from a wide range of video scenes and systems. The method is being implemented in a new end-to-end video quality monitoring tool that utilizes the Internet to communicate the low bandwidth features between the source and destination ends.
    
    
  • KEYWORDS: Video quality, in-service, reduced reference, subjective, objective, correlation
  
  
• Video Scaling Estimation Technique, NTIA Technical Memorandum TM-05-417, January, 2005.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: Digital video compression algorithms are being deployed that spatially stretch or shrink the video picture. Although small changes in spatial scaling are not usually noticeable to viewers, objective video quality measurement systems may be adversely impacted if the spatial scaling is not corrected. This report describes an algorithm that can be used to automatically measure the amount of spatial scaling present in a video system. This algorithm obtains satisfactory computational complexity by (1) separating the searches for horizontal & vertical scaling factors, (2) using image profiles rather than full images, and (3) using random rather than exhaustive searching techniques.
    
    
  • KEYWORDS: Calibration; objective; random search; spatial scaling; video quality
  
  
• A New Standardized Method for Objectively Measuring Video Quality, IEEE Transactions on Broadcasting, v. 50, n. 3, pp. 312-322, September, 2004.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full Reference Television (FR-TV) test. The NTIA General Model was the only video quality estimator that was in the top performing group for both the 525-line and 625-line video tests. As a result, the American National Standards Institute (ANSI) adopted the NTIA General Model and its associated calibration techniques as a North American Standard in 2003. The International Telecommunication Union (ITU) has also included the NTIA General Model as a normative method in two Draft Recommendations. This paper presents a description of the NTIA General Model and its associated calibration techniques. The independent test results from the VQEG FR-TV Phase II tests are summarized, as well as results from eleven other subjective data sets that were used to develop the method.
    
    
  • KEYWORDS: Video Quality, Image Quality, objective testing, subjective testing.
  
  
• The Impact of Monitor Resolution and Type on Subjective Video Quality Testing, NTIA Technical Memorandum TM-04-412, March, 2004.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: This document compares subjective video quality test results from a professional cathode ray tube (CRT) television monitor with that of a consumer liquid crystal display (LCD) video phone monitor. The CRT monitor supported the full ITU-R Recommendation BT.601 resolution (720 x 486) while the LCD monitor only supported Common Intermediate Format (CIF) resolution (352 x 288). The subjective results from the two tests are very similar, with the only significant difference being that the CIF monitor masks impairments that appear in only one of the two interlaced fields.
    
    
  • KEYWORDS: CIF; image quality; ITU-R Recommendation BT.601; monitor resolution; subjective testing; video quality
  
  
• Color Correction Matrix for Digital Still and Video Imaging Systems, NTIA Technical Memorandum TM-04-406, December, 2003.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: This document discusses a method for correcting inaccurate color output by digital still and video imaging systems. The method uses a known reference image together with a least-squares algorithm to estimate the optimal color channel mixing matrix that must be applied to the output images in order to correct their color inaccuracies. The techniques presented in this document will provide users of digital photography and video equipment with an automated tool for correcting color output. For instance, digital photography users currently may try to correct color distortions in their images by trial and error using photo editing software. However, these correction procedures are time consuming and subjective and do not normally allow for arbitrary mixing of the color channels. The automated color correction matrix computation presented in this document allows each color component in the corrected image (e.g., red) to be calculated as a linear summation of a DC component and all the color components (e.g., red, green, and blue) in the uncorrected image. Methods to correct non-linearities in the color response of digital imaging systems are also discussed.
    
    
  • KEYWORDS: calibration; camera; channel; chart; color; colorspace; component; correction; digital; matrix; non-linear; sRGB; video
  
  
• An Objective Method for Combining Multiple Subjective Data Sets, SPIE Video Communications and Image Processing Conference, Lugano, Switzerland, July, 2003.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subjective tests also have different rating scales, different words associated with these scales, and many other test variables that change from one laboratory to another (e.g., viewer expertise and criticality, cultural differences, physical test environments). Thus, it is very difficult to directly compare or combine results from two or more subjective experiments. The ability to compare and combine results from multiple subjective experiments would greatly benefit developers and users of video technology since standardized subjective data bases could be expanded upon to include new source material and past measurement results could be related to newer measurement results. This paper presents a subjective method and an objective method for combining multiple subjective data sets. The subjective method utilizes a large meta-test with selected video clips from each subjective data set. The objective method utilizes the functional relationships between objective video quality metrics (extracted from the video sequences) and corresponding subjective mean opinion scores (MOSs). The objective mapping algorithm, called the iterated nested least-squares algorithm (INLSA), relates two or more independent data sets that are themselves correlated with some common intermediate variables (i.e, the objective video quality metrics). We demonstrate that the objective method can be used as an effective substitute for the expensive and time consuming subjective meta-test.
    
    
  • KEYWORDS: single stimulus continuous quality evaluation (SSCQE), double stimulus continuous quality scale (DSCQS), comparison, correlation, video quality, image quality, subjective testing, objective testing.
  
  
• Comparing Subjective Video Quality Testing Methodologies, SPIE Video Communications and Image Processing Conference, Lugano, Switzerland, July, 2003.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the quality or change in quality between two video streams (reference and impaired). Others are single stimulus where viewers rate the quality of just one video stream (the impaired). Two examples of the former are the double stimulus continuous quality scale (DSCQS) and double stimulus comparison scale (DSCS). An example of the latter is single stimulus continuous quality evaluation (SSCQE). Each subjective test methodology has claimed advantages. For instance, the DSCQS method is claimed to be less sensitive to context (i.e., subjective ratings are less influenced by the severity and ordering of the impairments within the test session). The SSCQE method is claimed to yield more representative quality estimates for quality monitoring applications. This paper considers data from six different subjective video quality experiments, originally performed with SSCQE, DSCQS and DSCS methodologies. A subset of video clips from each of these six experiments were combined and rated in a secondary SSCQE subjective video quality test. We give a method for postprocessing the secondary SSCQE data to produce quality scores that are highly correlated to the original DSCQS and DSCS data. We also provide evidence that human memory effects for time-varying quality estimation seem to be limited to about 15 seconds.
    
    
  • KEYWORDS: single stimulus continuous quality evaluation (SSCQE), double stimulus continuous quality scale (DSCQS), double stimulus comparison scale (DSCS), correlation, video quality, image quality, subjective testing, picture quality
  
  
• Video Quality Measurement PC User's Manual, November, 2002.
  
  
  • AUTHORS: Margaret Pinson, Stephen Wolf, Phillip G. Austin, and Andrea Allhands
    
    
  • ABSTRACT: The purpose of this handbook is to provide a user's manual for the video quality measurement (VQM) PC tool. This VQM software tool performs user-friendly interactive processing of video files. Program VQM runs under the WINDOWS® operating system and provides two methods for making video quality measurements. In the first method, the user must have original (i.e., reference) and processed (i.e., impaired) video clips stored on disk. In the second method, the user must have a set of original video clips stored on disk. Here the VQM tool can be used to construct a test video sequence (TVS) from this set of original clips, play the constructed original TVS through a video system under test, capture the processed TVS output from the video system, and parse the clips in the TVS for subsequent video quality analyses. Program VQM compares the video sequence that has been processed by the video system under test to the original video sequence through two main steps. First, program VQM calibrates the processed video sequence to remove systematic differences between the original and processed, such as spatial and temporal shifts. Second, program VQM estimates and reports the perceived quality of the processed video using one of five video quality models. Quality estimates are reported on a default scale of zero to one, where zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level. The VQM PC tool is written in such a manner that it may be easily updated to incorporate future improvements in video calibration and quality analysis. Users are encouraged to email their suggestions for improving the VQM PC tool to vqm@its.bldrdoc.gov.
    
    
  • KEYWORDS: video quality metrics, subjective, objective, correlation, spatial, temporal, calibration, registration, user manual, handbook, PC, Windows, software, video files
  
  
• Video Quality Measurement Techniques, National Telecommunications and Information Administration (NTIA) Report 02-392, June, 2002.
  
  
  • AUTHORS: Stephen Wolf and Margaret Pinson
    
    
  • ABSTRACT: Objective metrics for measuring digital video performance are required by Government and industry for specification of system performance requirements, comparison of competing service offerings, service level agreements, network maintenance, and optimization of the use of limited network resources such as transmission bandwidth. To be accurate, digital video quality measurements must be based on the perceived quality of the actual video being received by the users of the digital video system rather than the measured quality of traditional video test signals (e.g., color bar). This is because the performance of digital video systems is variable and depends upon the dynamic characteristics of both the original video (e.g., spatial detail, motion) and the digital transmission system (e.g., bit rate, error rate). The goal of this report is to provide a complete description of the ITS video quality metric (VQM) algorithms and techniques. The ITS automated objective measurement algorithms provide close approximations to the overall quality impressions, or mean opinion scores, of digital video impairments that have been graded by panels of viewers.
    
    
  • KEYWORDS: video, quality, models, metrics, features, parameters, objective, subjective, correlation, reduced-reference, television, videoconferencing, root cause analysis, spatial information (SI), temporal information (TI), impairments, blocking, blurring, frame dropping, peak-signal-to-noise ratio (PSNR), video calibration, spatial registration, temporal registration, gain, contrast, level offset, brightness
  
  
• Video Quality Measurement User Manual, National Telecommunications and Information Administration (NTIA) Handbook 02-01, February, 2002.
  
  
  • AUTHORS: Margaret Pinson and Stephen Wolf
    
    
  • ABSTRACT: The purpose of this handbook is to provide a user's manual for the video quality metric (VQM) tool. The VQM software tool performs automated batch processing of video files. Program VQM runs under the UNIX operating system and uses a control file to specify the exact video quality measurement procedures that are to be performed. All results are emailed to the user. Program VQM compares the video sequence that has been processed by the video system under test to the original video sequence through two main steps. First, program VQM calibrates the processed video sequence to remove systematic differences between the original and processed, such as spatial and temporal shifts. Second, program VQM estimates and reports the perceived quality of the processed video using one of five video quality models. Quality estimates are reported on a scale of zero to one, where zero means that no impairment is visible and one means that the video clip has reached the maximum impairment level.
    
    
  • KEYWORDS: video quality metrics, subjective, objective, correlation, spatial, temporal, calibration, registration, user manual, handbook, UNIX, software, video files
  
  
• On the Impact of Policing and Rate Guarantees in Diff-Serv Networks: A Video Streaming Application Perspective, SIGCOMM 2001 (Association for Computing Machinery, Special Interest Group on Data Communications), August, 2001.
  
  
  • AUTHORS: Wael Ashmawi, Roch Guerin, Stephen Wolf, and Margaret Pinson
    
    
  • ABSTRACT: Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore how the policing actions and associated rate guarantees provided by the Expedited Forwarding (EF) translate into perceived benefits for applications that are the presumed users of such enhancements. Specifically, we focus on video streaming applications that arguably have relatively strong service quality requirements, and which should, therefore, stand to benefit from the availability of some form of enhanced service. Our goal is to gain a better understanding of the relation that exists between application level quality measures and the selection of the network level parameters that govern the delivery of the guarantees that an EF based service would provide. Our investigation, which is experimental in nature, relies on a number of standard streaming video servers and clients that have been modified and instrumented to allow quantification of the perceived quality of the received video stream. Quality assessments are performed using a Video Quality Measurement tool based on the ANSI objective quality standard. Measurements were made over both a local Diff-Serv testbed and across the QBone, a QoS enabled segment of the Internet2 infrastructure. The paper reports and analyzes the results of those measurements.
  • KEYWORDS: streaming video quality metrics, subjective, objective, correlation, Internet2, differentiated services, expedited forwarding, quality of service
  
  
• The Relationship Between Performance and Spatial-Temporal Region Size for Reduced-Reference, In-Service Video Quality Monitoring Systems, SCI / ISAS 2001 (Systematics, Cybernetics, and Informatics / Information Systems Analysis and Synthesis), July, 2001.
  
  
  • AUTHORS: Stephen Wolf and Margaret H. Pinson
    
    
  • ABSTRACT: This paper presents objective-to-subjective correlation results for a reduced-reference, in-service, video quality monitoring system. This reduced-reference system utilizes quality parameters that are computed by comparing features extracted from spatial-temporal (S-T) regions of the input video stream with identical features extracted from the output video stream. The amount of reduced-reference information that is required to compute the quality parameters is inversely related to the size of the S-T region. Smaller amounts of reference information (i.e., larger S-T regions) are desired since less transmission or storage bandwidth is required for the reference information. However, objective-to-subjective correlation drops off if the S-T region size becomes too large. In this paper we examine the tradeoffs between objective-to-subjective correlation results and S-T region size. Correlation results for S-T region sizes from 8 vertical lines x 8 horizontal pixels x 2 video frames to 128 x 128 x 24 are presented. These results utilized a total of nine subjectively rated data sets that span an extremely wide range of bit rates and compression techniques. Thus, designers of television video systems as well as Internet video streaming systems may use the results.
    
    
  • KEYWORDS: video quality metrics, subjective, objective, correlation, spatial, temporal, in-service, compression, reduced-reference, sub-region
  
  
• Spatial-Temporal Distortion Metrics for In-Service Quality Monitoring of Any Digital Video System, SPIE International Symposium on Voice, Video, and Data Communications, Boston, MA, September 11-22, 1999.
  
  
  • AUTHORS: Stephen Wolf and Margaret H. Pinson
    
    
  • ABSTRACT: Many organizations have focused on developing digital video quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for in-service quality monitoring. The Institute for Telecommunication Sciences (ITS) has developed spatial-temporal distortion metrics that meet all of these requirements. These objective metrics are described in detail and have a number of interesting properties, including utilization of 1) spatial activity filters which emphasize long edges on the order of 1/5 degree while simultaneously performing large amounts of noise suppression, 2) the angular direction of the spatial gradient, 3) spatial-temporal compression factors of at least 384:1 (spatial compression of at least 64:1 and temporal compression of at least 6:1, and 4) simple perceptibility thresholds and spatial-temporal masking functions. Results are presented that compare the objective metric values with mean opinion scores from a wide range of subjective data bases spanning many different scenes, systems, bit-rates, and applications.
    
    
  • KEYWORDS: video quality metrics, subjective, objective, correlation, spatial, temporal, in-service, compression
  
  
• Medium Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences, ANSI T1A1 contribution number T1A1.5/99-205, May, 1999.
  
  
  • AUTHORS: Maragaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: This contribution presents new techniques for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution considers "variable alignment," defined as an alignment process that can assign a unique alignment offset, or delay, to every output video frame in a given output video sequence. To achieve the bandwidth reduction necessary for in-service measurements, the variable alignment techniques presented in this contribution are applied to subsampled video images. We have found that a sub-sampling factor of 128 to 1 produces useful variable alignment results. Regardless of the sub-sampling factor used (including sub-sampling factors of 1), there is always a possibility of misalignment. The probability of misalignment increases as the amount of scene motion decreases and the video distortion present in the output increases. Therefore, it has been necessary to implement artificial intelligence techniques that examine the set of most probable alignments produced by the signal processing portion of the algorithm. All of the techniques presented in this contribution can also be applied to whole field correlation algorithms such as those given in ANSI T1.801.04.
    
    
  • KEYWORDS: video delay, temporal alignment, medium bandwidth features
  
  
• Low Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences, ANSI T1A1 contribution number T1A1.5/99-204, May, 1999.
  
  
  • AUTHORS: Margaret H. Pinson and Stephen Wolf
    
    
  • ABSTRACT: This contribution presents a new technique for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution is limited to "constant alignment," defined as an alignment process that computes the same fixed alignment offset, or delay, for every output video frame in a given output video sequence. Constant alignment can (1) serve as a starting point for variable alignment techniques, where each output frame can have a unique alignment offset, or delay, and (2) be used to align low frame rate output video sequences in preparation for measuring video quality parameters. The constant alignment technique that is presented is applicable for real-time in-service monitoring since it utilizes a set of computationally efficient low bandwidth features that are extracted from the input and output video streams.
    
    
  • KEYWORDS: video delay, temporal alignment, low bandwidth features
  
  
• In-Service Performance Metrics for MPEG-2 Video Systems, Made to Measure 98 - Measurement Techniques of the Digital Age Technical Seminar, jointly sponsored by the International Academy of Broadcasting (IAB), the International Telecommunications Union (ITU), and the Technical University of Braunschweig (TUB), Montreux, Switzerland, November 12-13, 1998.
  
  
  • AUTHORS: Stephen Wolf and Margaret H. Pinson
    
    
  • ABSTRACT: With the advent of new digital video systems that utilize compression to achieve a savings in transmission or storage bandwidth, the quality of the received output video can be dependent not only upon the inherent spatial and temporal information content of the input video but also upon the dynamic variability of the digital communications channel. Therefore, out-of-service quality measurements using video test signals or scenes may not relate at all to the resultant received quality of actual program material. Furthermore, traditional in-service quality measurements made by injecting test signals into the non-visible portion of the video signal (e.g., the vertical interval in the NTSC or PAL video standard) are not applicable to modern digital video systems. Thus, a new method is required to make in-service video quality measurements on actual program material. This paper describes a new test instrument for measuring the quality of a video transmission or storage system where the input and output of the system may be spatially separated, and when there is no a priori knowledge of the input video. The test instrument makes continuous quality measurements by (1) extracting statistics from sequences of processed input and output video frames, (2) communicating these extracted statistics between the input and the output ends using an ancillary-data channel of arbitrary bandwidth, (3) computing individual video quality parameters from the communicated statistics that are indicative of the various perceptual aspects of video quality (e.g., spatial, temporal, color), and (4) calculating a composite video quality metric by combining the individual video quality parameters. The test instrument makes coarser quality measurements (coarser in the sense that the extracted statistics come from larger spatial-temporal regions) when smaller capacity ancillary-data channels are available and finer quality measurements when larger capacity ancillary-data channels are available. The design goal for the test instrument is to make the most accurate in-service video quality measurements given the available ancillary-data-channel bandwidth (mobile telephone connections, modem connections over the Public Switched Telephone Network (PSTN), Internet connections, Local Area Network (LAN) connections, satellite connections, cable connections, etc.).
    
    
  • KEYWORDS: MPEG-2, video, quality, performance, metrics, measures, subjective, objective, correlation, parameters, features, spatial, temporal, color, test, instrument
  
  
• Perceptual Effects of Noise in Digital Video Compression, 140th SMPTE Technical Conference, Pasadena, CA, October 28-31, 1998.
  
  
  • AUTHORS: Charles Fenimore and John Libert (NIST), Stephen Wolf (NTIA/ITS)
    
    
  • ABSTRACT: We present results of subjective viewer assessment of video quality of MPEG-2 compressed video containing wide-band Gaussian noise. The video test sequences consisted of seven test clips (both classical and new materials) to which noise with a peak-signal-to-noise-ratio (PSNR) of from 28 dB to 47 dB was added. We used software encoding and decoding at five bit-rates ranging from 1.8 Mb/s to 13.9 Mb/s. Our panel of 32 viewers rated the difference between the noisy input and the compression-processed output. For low noise levels, the subjective data suggests that compression at higher bit-rates can actually improve the quality of the output, effectively acting like a low-pass filter. We define an objective and a subjective measure of scene criticality (the difficulty of compressing a clip) and find the two measures correlate for our data. For difficult-to-encode material (high criticality), the data suggest that the effects of compression may be less noticeable at mid-level noise, while for easy-to-encode video (low criticality), the addition of a moderate amount of noise to the input led to lower quality scores. This suggests that either the compression process may have reduced noise impairments or a form of masking may occur in scenes that have high levels of spatial detail.
    
    
  • KEYWORDS: Noise, MPEG-2, perception, objective, subjective, metrics, correlation, video, quality, scene, criticality
  
  
• "Development of Opinion-Based Audiovisual Quality Models for Desktop Video-Teleconferencing, 6th IEEE International Workshop on Quality of Service, Napa, California, May 18-20, 1998.
  
  
  • AUTHORS: Coleen Jones and D.J. Atkinson
    
    
  • ABSTRACT: This paper discusses the analysis of an audiovisual desktop video-teleconferencing subjective experiment conducted at the Institute for Telecommunication Sciences. Objective models of the individual audio and video quality are presented. Also discussed is an objective model of the audio-visual quality based upon the results of the individual objective audio and video quality models. Finally, a subjective model of the audiovisual quality based upon users' ratings of the audio and video quality is discussed.
    
    
  • KEYWORDS: Audio, video, audio-visual, quality, objective, subjective, models, correlation, video teleconferencing
  
  
• Objective and Subjective Measures of MPEG Video Quality, 139th SMPTE Technical Conference, New York City, November 21-24, 1997.
  
  
  • AUTHORS: Stephen Wolf, Margaret H. Pinson, and Arthur A. Webster (NTIA/ITS), Gregory W. Cermak and E. Paterson Tweedy (GTE Laboratories)
    
    
  • ABSTRACT: In 1996, the American National Standards Institute (ANSI) adopted ANSI T1.801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI T1.801.03 were selected based on an extensive multilaboratory quality assessment study that included video systems from bit rates of 64 kbit/sec to 45 Mbit/sec and video test scenes that spanned a wide range of spatial and temporal coding difficulties. The set of objective video quality measurements effectively accounted for subjective judgments by human viewers. While 25 video systems were tested, this multilaboratory study did not include MPEG video systems, and did not cover any bit rates between 1.6 and 10 Mbit/sec. This paper presents the results from two MPEG studies designed to fill in the bit-rate gap in the previous multilaboratory study. In these studies, we concentrated on bit rates from 1.5 - 8.3 Mbit/sec and examined the performance of MPEG 1 and MPEG 2 codecs (coder-decoders) specifically. The effectiveness of the ANSI standard objective video quality metrics was examined for these bit rates and coding technologies. Our analysis revealed that the objective video quality metrics primarily measure four principal components of video quality: added edges, lost edges, added motion, and lost motion; we found that parameters selected from these principal components can be used as effective predictors of subjective quality ratings for entertainment video systems.
    
    
  • KEYWORDS: video, quality, MPEG, MPEG-1, MPEG-2, coding, compression, subjective, objective, correlation, metrics
  
  
• Measuring the End-to-End Performance of Digital Video Systems, IEEE Trans. Broadcast., Vol. 43, No. 3, pp. 320-328, September, 1997.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Significant research and development efforts by industry and government laboratories were brought to fruition in 1996 with the approval of American National Standard (ANSI) T1.801.03 entitled "American National Standard for Telecommunications - Digital Transport of One-Way Video Signals - Parameters for Objective Performance Assessment." This standard provides a set of objective parameters that have consistently demonstrated high correlation levels with subjective evaluations of digital video impairments. The parameters are technology-independent and may be used to measure the performance of a wide range of digital video compression, storage, and transmission systems. This paper presents an overview of the ANSI T1.801.03 parameters and summarizes other relevant standards activities and contributions.
    
    
  • KEYWORDS: Video, digital, quality, performance, objective, subjective, correlation, standards, ANSI, parameters
  
  
• Objective and Subjective Measures of MPEG Video Quality, ANSI T1A1 contribution number T1A1.5/96-121, October 28, 1996.
  
  
  • AUTHORS: Stephen Wolf, Margaret Pinson, Arthur Webster (NTIA/ITS), Greg Cermak and E. Paterson Tweedy (GTE Laboratories)
    
    
  • ABSTRACT: Presents an in-depth analysis and discussion of the results from applying the ANSI T1.801.03-1996 objective video quality metrics to subjectively rated MPEG-1 and MPEG-2 video test scenes. The objective metrics presented in ANSI T1.801.03-1996 (American National Standard for Telecommunications - Digital Transport of One-Way Video Signals - Parameters for Objective Performance Assessment) were able to account for 90% of the subjective information that could be captured considering the level of measurement error present in the subjective and objective data sets. By contrast, peak signal to noise ratio (PSNR), a traditional objective metric, was only able to account for 21% of the subjective information.
    
    
  • KEYWORDS: video, quality, MPEG, MPEG-1, MPEG-2, coding, compression, subjective, objective, correlation, metrics
  
  
• Measuring Digital Video Transmission Channel Gain, Level Offset, Active Video Shift, and Video Delay, ANSI T1A1 contribution number T1A1.5/96-110, May 31, 1996.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Presents a computerized search method for determining the gain, level offset, active video shift, and video delay of a digital video transmission channel. The method uses digitized input and output NTSC video fields that have been time tagged with SMPTE time code. The primary applications of the method are (1) to produce estimates of channel gain, level offset, and active video shift when data from ANSI T1.801.03-1996 calibration test patterns are not available, and (2) to provide a dynamic method to measure these quantities in conjunction with video delay.
    
    
  • KEYWORDS: video, gain, level offset, shift, delay, calibration, dynamic, variable, digital, coding, compression
  
  
• Exploring B-ISDN Performance Interactions: Selected Experiments and Results, NTIA Report 96-329, April, 1996.
  
  
  • AUTHORS: D.J. Atkinson
    
    
  • ABSTRACT: This report describes experiments conducted to explore the user-information transfer performance of the broadband integrated services digital network (B-ISDN), the emerging infrastructure for the global information age. These performance experiments include studying the effect of physical layer transmission performance on asynchronous transfer mode (ATM) cell transfer performance, ATM performance in relationship to network topology, and the impact of B-ISDN performance on video quality. A tool to help study these performance issues, a B-ISDN network emulator, is described, including its validation. The emulator incorporates a novel model for transmission impairments, enabling performance interactions among the B-ISDN protocol layers to be studied based on relevant International Telecommunication Union - Telecommunication Standardization Sector (ITU-T) Recommendations and American National Standards.
    
    
  • KEYWORDS: asynchronous transfer mode, ATM, B-ISDN, broadband, emulation, measurement, network, performance, SONET, standards, synchronous optical network
  
  
• Tape Duplication and Traceability Process for ANSI T1.801.02-1996 (Terms and Definitions), ANSI T1A1 contribution number T1A1.5/95-158, December 14, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.02-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Performance Terms, Definitions, and Examples).
    
    
  • KEYWORDS: video, scenes, examples, impairments, digital, coding, compression, tape, traceability, duplication, guidelines, conferencing, artifacts
  
  
• Tape Duplication and Traceability Process for ANSI T1.801.01-1996 (Test Scenes), ANSI T1A1 contribution number T1A1.5/95-157, December 14, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.01-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Video Test Scenes for Subjective and Objective Performance Assessment).
    
    
  • KEYWORDS: video, scenes, testing, impairments, digital, coding, compression, tape, traceability, duplication, guidelines, conferencing
  
  
• Field Vs. Frame Calculations for Visual Channel Delay, ANSI contribution number T1A1.5/95-152, December 11, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: This contribution recommends that calculations of visual channel delay be based on fields rather than frames for an interlaced video system such as NTSC. Using fields rather than frames will double the resolution of the measurement without significantly increasing the complexity and will overcome certain measurement anomalies that might result from calculations based on frames.
    
    
  • KEYWORDS: video, delay, NTSC, field, frame, rate, variable, constant
  
  
• An Analysis Technique for Detecting Temporal Edge Noise, ANSI contribution number T1A1.5/95-105, January 9, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Presents an objective analysis technique that can be used for detecting temporal edge noise in an output video sequence that has been digitally compressed. Temporal edge noise is defined in ANSI T1.801.02-1996 as "A form of edge busyness characterized by time-varying sharpness to edges of objects." The technique uses the Fourier transform to perform a comparative analysis of the frequencies present in the time history of the spatial information (SI) features measured from the input and output video (SI features are defined in ANSI T1.801.03-1996). The presence of temporal edge noise in the output video is shown to result in high frequency information being added to the time history of the SI feature.
    
    
  • KEYWORDS: video, scene, performance, testing, objective, quality, edge, noise, digital, impairments, temporal
  
  
• Objective Measures for Detecting Digital Tiling, ANSI contribution T1A1.5/95-104, January 9, 1995.
  
  
  • AUTHORS: Dwight Melcher, Stephen Wolf
    
    
  • ABSTRACT: Describes an algorithm for (1) quantifying the spatial gradients or edges in an image as a function of angle or orientation, (2) extracting low bandwidth features from these input and corresponding output spatial gradient images, and (3) deriving objective video quality metrics from these low bandwidth features that quantify the amount of tiling (i.e., blocking) and blurring in the output image. Examples are given for images that have been compressed by MPEG and video teleconferencing codecs.
    
    
  • KEYWORDS: video, scene, performance, testing, objective, quality, blocking, tiling, digital, compression, impairments, spatial, MPEG, teleconferencing, codecs
  
  
• An In-depth Analysis of the P6 Lost Motion Energy Parameter, ANSI contribution number T1A1.5/95-103, January 9, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Provides an in-depth analysis of the behavior of the P6 video performance parameter, a parameter the measures the motion energy that is lost when a video scene is transmitted through a digital video compression system. Objective to subjective correlation results are given for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). This contribution provides experimental evidence that video teleconferencing users do not significantly downgrade quality ratings for frame rates of at least 10 to 15 frames per second.
    
    
  • KEYWORDS: video, scene, performance, objective, subjective, quality, correlation, frame rate, dropped frames, motion, noise, temporal, impairments, models, codecs, compression, teleconferencing
  
  
• Assessing the Composite Performance of a Hypothetical Reference Circuit (HRC) for an Ensemble of Source Material, ANSI contribution number T1A1.5/95-102, January 9, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: This contribution discusses the advantages and disadvantages of two methods for obtaining a composite rating of Hypothetical Reference Circuit (HRC) performance for an ensemble of source material. The term HRC refers to a specific realization of a video transmission system that may include coders, digital transmission circuits, decoders, and even analog processing of the video signal. In the first method, subjects are asked to separately observe and rate each of several HRC-scene combinations, and the separate ratings are then averaged over subjects and scenes to produce a composite rating for each HRC. In the second method, subjects are asked to rate each HRC based on a single observation period in which all the scenes are presented; the individual subject ratings are then averaged to produce a composite rating for each HRC. This contribution evaluates the two methods in light of accepted industry practice and statistical considerations. It is shown that the first method is far more widely used and offers substantial statistical advantages over the second.
    
    
  • KEYWORDS: video, system, performance, testing, objective, subjective, quality
  
  
• Two Criteria for Video Test Scene Selection, ITU-T (Telecommunications Standardization Sector) Temporary Document 35-E, Working Party 2, Study Group 12, Question 22, Geneva, December 2-5, 1994.
  
  
  • AUTHORS: Arthur Webster
    
    
  • ABSTRACT: Describes two objective measurements that can be used to quantify the spatial and temporal information content of a video test scene. Test scene coding difficulty (and hence video quality) often depend on the spatial and temporal information complexity of the source video. The metrics presented here can be used to assist in the selection of appropriate video test scenes for subjective and objective performance tests of digital video compression systems.
    
    
  • KEYWORDS: video, scene, selection, subjective, objective, performance, spatial, temporal, complexity, information
  
  
• Analysis of T1A1.5 Subjective and Objective Test Data, October 3, 1994.
  
  
  • AUTHORS: Coleen Jones, Ned Crow, Stephen Wolf, Arthur Webster
    
    
  • ABSTRACT: Presents detailed intra-laboratory (within laboratory) and inter-laboratory (between laboratories) analysis of variance (ANOVA) results for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). The ANOVA results for the subjective test data were obtained by applying the techniques in ANSI contribution T1A1.5/94-128. A correlation analysis is given of the objective parameters presented in ANSI contributions T1A1.5/93-153, T1A1.5/93-152 (corrections given in T1A1.5/94-110), and T1A1.5/94-102. In addition to the detailed ANOVA results, plots are presented which compare the subjective data from the different laboratories.
    
    
  • KEYWORDS: video, quality, performance, objective, subjective, correlation, ANOVA, results, digital, compression, systems, conferencing
  
  
• Status of NTIA's Analysis of Subjective Data, ANSI contribution number T1A1.5/94-133, July 18, 1994.
  
  
  • AUTHORS: Coleen Jones, Ned Crow
    
    
  • ABSTRACT: Presents results from the analysis of variance (ANOVA) of the subjective opinion scores for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). This contribution includes the analysis of the three teams (green, red, and orange) for two of the three laboratories involved in the testing. The analysis was performed as discussed in contribution T1A1.5/94-128 ("Methods for Analysis of Inter-laboratory Video Performance Standard Subjective Test Data"). The ANOVAs showed that all main effects and interactions were significant for all teams within the two laboratories.
    
    
  • KEYWORDS: video, quality, performance, objective, subjective, experiment, statistical, analysis, ANOVA
  
  
• The Correlation of Traditional Bandwidth and Signal-to-Noise Ratio Parameters to Subjective Data, ANSI contribution number T1A1.5/94-132, July 18, 1994.
  
  
  • AUTHORS: Coleen Jones
    
    
  • ABSTRACT: This contribution presents objective to subjective correlation results for the traditional analog measurements of bandwidth (i.e., frequency response) and signal-to-noise ratio (SNR) for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec).
    
    
  • KEYWORDS: video, quality, performance, objective, subjective, SNR, frequency, response, bandwidth
  
  
• Methods for Analysis of Inter-laboratory Video Performance Standard Subjective Test Data, ANSI contribution number T1A1.5/94-128, March 28, 1994.
  
  
  • AUTHORS: Edwin L. Crow
    
    
  • ABSTRACT: Presents in detail the ANOVA methods for analysis of the subjective T1A1 video test data, especially with regard to any systematic effect a laboratory may have on the Mean Opinion Score (MOS) of any given Hypothetical Reference Circuit (HRC). The term HRC refers to a specific realization of a video transmission system that may include coders, digital transmission circuits, decoders, and even analog processing of the video signal. The experimental variables of the T1A1 subjective test plan may be summarized as follows: (1) 3 laboratories, X, Y, Z, (2) 25 HRCs, 1, 2,..., 25, (3) 25 scenes, a, b,..., y, (4) 625 HRC-scene combinations, or test combinations, (5) 30 accepted viewers in each lab, screened from about 36 initial viewers, some of whom may not pass a consistency check, broken into 3 teams of 10 each, (6) 3 sets of videotapes - Red (R), Green (G), Orange (O), each set of 4 tapes to be viewed by a corresponding team in each lab, each set including 10 HRCs (thus overlapping slightly), (7) 4 subteams within each team since each viewing session is limited to at most 3 viewers, (8) 4 sessions for each subteam to view the 4 tapes, each tape (and session) being limited to about 32 minutes, (9) 9 types (1, 2,..., 9) into which the 25 HRCs are classified, 1 to 4 in each type, (10) 5 content categories (A,B,C,D,E) of the 25 scenes, 3 to 6 in each category, (11) 5 possible ratings of test combination scene impairment by viewers on voting forms ranging from Imperceptible to Very Annoying, which will be translated into 5,4,3,2,1 in the data reduction. Test combinations are ordered on the tapes by a restricted randomization, that subteams are selected at random from the total viewers available (from a specified type of population), and that the four tapes are presented to the corresponding four subteams in random permutation orders.
    
    
  • KEYWORDS: subjective, video, quality, ANOVA, statistical, analysis, inter-laboratory
  
  
• Corrections and Extensions to T1A1.5/93-152, ANSI contribution number T1A1.5/94-110, January 17, 1994.
  
  
  • AUTHORS: Stephen Wolf, Margaret Pinson
    
    
  • ABSTRACT: Contribution T1A1.5/93-152 summarized the methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image that were submitted prior to conducting the T1A1 subjective experiment (this experiment collected 625 mean opinion scores - i.e., 25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). This contribution presents (1) one minor correction to the recommended value for the fraction above threshold in contribution T1A1.5/93-152, and (2) a method for estimating the video delay uncertainty of the automated time alignment algorithm presented in section 3 of contribution T1A1.5/93-152 (non-zero video delay uncertainty may result when dynamic time warping, or variable video delay, is present in the video transmission system, or when there is a substantial number of dropped video frames), (3) a method for using this video delay uncertainty in the computation of the parameters presented in T1A1.5/93-152, and (4) an improved motion spike detector that could be used for computing parameters p10 and p11 in T1A1.5/93-152.
    
    
  • KEYWORDS: objective, video, quality, testing, performance, metrics, measurements, parameters, compression, coding, artifacts, variable, delay, motion, dropped, frames
  
  
• VTC Hypothetical Reference Circuit Signal to Noise Ratio Measurements, ANSI contribution number T1A1.5/94-103, January 17, 1994.
  
  
  • AUTHORS: Arthur Webster, Coleen Jones
    
    
  • ABSTRACT: Discusses the method of measurement used to obtain the signal to noise ratio (SNR) values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The signal to noise ratio (SNR) was measured using a traditional analog test waveform, in particular, the 55 IRE flat field test signal that was on the standard video test scene tape.
    
    
  • KEYWORDS: objective, video, quality, performance, specifications, SNR, measurements, noise, conferencing
  
  
• VTC Hypothetical Reference Circuit Bandwidth Measurements, January 17, 1994.
  
  
  • AUTHORS: Coleen Jones
    
    
  • ABSTRACT: Discusses the method of measurement used to obtain the frequency response or bandwidth values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The bandwidth was measured using a traditional analog test waveform, in particular, a static zone plate (i.e., a two dimensional swept frequency waveform) that was on the standard video test scene tape.
    
    
  • KEYWORDS: objective, video, quality, performance, specifications, bandwidth, measurements, frequency, zone, plate, response, conferencing
  
  
• User-Oriented Measures of Telecommunication Quality, IEEE Communications Magazine, January, 1994.
  
  
  • AUTHORS: Neal Seitz, Stephen Wolf, Stephen Voran, Randy Bloomfield
    
    
  • ABSTRACT: Discusses the standardization of user-oriented, technology-independent measures of telecommunication service quality, including voice, video, and data performance standards. Standards committee work has progressed in three broad phases. In the first phase, participants defined the basic concepts that underline the user-oriented approach to telecommunication quality assessment. In the second phase, participants developed a set of generic user-oriented quality measures, and applied these generic measures in deriving technology-specific performance parameters and measurement methods for packet-switched networks and integrated services digital networks (ISDNs). In the third phase, participants have developed user-oriented quality measures for video and voice communications. User-perceivable video and voice quality impairments are quantified by objective measures chosen for their correlation with carefully collected and numerically quantified human reactions to the transmitted images and sounds.
    
    
  • KEYWORDS: telecommunication, quality, performance, metrics, data, voice, video, standards, objective, subjective, correlation, perception-based, ISDN, ATM
  
  
• Methods of Measurement for Two Objective Video Quality Parameters Based on the Fourier Transform, ANSI contribution number T1A1.5/93-153, November 8, 1993.
  
  
  • AUTHORS: Arthur Webster
    
    
  • ABSTRACT: Summarizes detailed methods of measurement for two objective video quality parameters based on the Fourier transform image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this experiment collected 625 mean opinion scores - i.e., 25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). The video quality parameters presented here have demonstrated strong correlation to subjective evaluations of the video and can be used for in-service as well as out-of-service tests since low bit-rate features are extracted and compared from the input and corresponding output video images.
    
    
  • KEYWORDS: objective, video, quality, performance, testing, parameters, features, compression, digital, Fourier, spatial, frequencies, measurements, metrics, correlation, subjective, in-service, out-of-service
  
  
• A Summary of Methods of Measurement for Objective Video Quality Parameters Based on the Sobel Filtered Image and the Motion Difference Image, ANSI contribution number T1A1.5/93-152, November 8, 1993.
  
  
  • AUTHORS: Stephen Wolf, Margaret Pinson, Coleen Jones, Arthur Webster
    
    
  • ABSTRACT: Summarizes detailed methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this experiment collected 625 mean opinion scores - i.e., 25 test scenes passed through 25 different video transmission systems that ranged in bit rate from 64 kb/sec to 45 Mb/sec). The video quality parameters presented here have demonstrated strong correlation to subjective evaluations of the video and can be used for in-service as well as out-of-service tests since low bit-rate features are extracted and compared from the input and corresponding output video images.
    
    
  • KEYWORDS: objective, video, quality, performance, testing, parameters, features, compression, digital, Sobel, motion, spatial, temporal, measurements, metrics, correlation, subjective, in-service, out-of-service
  
  
• Proposed Framework for Subjective Audiovisual Testing, ANSI contribution number T1A1.5/93-151, November 8, 1993.
  
  
  • AUTHORS: Stephen Voran, Stephen Wolf
    
    
  • ABSTRACT: Describes five basic tests that can be used to examine how perceived audio quality and perceived video quality combine to create a perceived level of composite audiovisual quality. These tests provide a logical and important first step toward understanding the components of subjective audiovisual quality. Two of the five tests could provide information about the interactions between perceived audio and video quality. The five resulting tests are summarized in tabular form.
    
    
  • KEYWORDS: subjective, audio, video, audiovisual, quality, testing, performance, interactions, framework
  
  
• Real-Time Measurement System for ITS Video Quality Parameters, ANSI contribution number T1A1.5/93-105, August 9, 1993.
  
  
  • AUTHORS: Coleen Jones
    
    
  • ABSTRACT: Discusses a real-time personal computer-based system for characterizing spatial (e.g., blurring) and temporal (e.g., jerky motion) distortions in a compressed digital video system. Comparisons with a high quality non real-time laboratory version of the same image processing algorithms demonstrates that measurement system gain and frequency response must be calibrated to assure repeatability of objective performance parameter results.
    
    
  • KEYWORDS: objective, video, quality, parameters, spatial, temporal, digital, compression, real-time, performance, measurement, system, PC-based, image, processing, noise, gain, frequency, response, Sobel, motion
  
  
• An Objective Technique for Assessing Video Impairments, IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, May, 1993.
  
  
  • AUTHORS: Stephen Voran, Stephen Wolf
    
    
  • ABSTRACT: Measurements that quantify perceptual video attributes in both the spatial and temporal domains are extracted from the video that has undergone digital coding and compression. These measurements are then used to compute a single score that quantifies the perceptual impact of the impairments present in the video sequence. This objective score is well-correlated (r=.92) with impairment assessments made by human viewers. A collection of 36 video scenes with a wide range of spatial and temporal complexity were passed through 28 video systems covering the range of compression from 45 Mbs/sec to 56 kbs/sec, including bit errors introduced into the digital transmission channel.
    
    
  • KEYWORDS: objective, subjective, correlation, video, quality, parameters, models, spatial, temporal, digital, compression, performance, assessment, conferencing, bit, errors
  
  
• Objective Performance Parameters for NTSC Video at the DS3 Rate, ANSI contribution number T1A1.5/93-060, April 28, 1993.
  
  
  • AUTHORS: Stephen Wolf, Arthur Webster
    
    
  • ABSTRACT: Summarizes objective video quality measurement results (and their correlation to subjective quality) for the subjective tests of contribution quality 45 Mbs/sec systems (i.e., studio to studio systems) conducted in 1990 by ANSI committee T1Y1.1. Expert viewers for this demanding application are shown to be about twice as critical as naive viewers for the same amount of video impairment. The video performance of particular DS3 systems is shown to vary depending upon the source scene, a somewhat surprising result considering the limited amount of compression introduced by these high bit-rate systems. The presence of noise impairments that have structure (e.g., noise lines) are shown to be more irritating to viewers than unstructured noise (i.e., noise that is randomly distributed throughout the picture area).
    
    
  • KEYWORDS: objective, subjective, video, quality, parameters, spatial, temporal, digital, compression, delay, performance, measurement, 45 Mbs/sec, DS3, correlation, expert, naive, viewers, models
  
  
• An objective video quality assessment system based on human perception, SPIE Human Vision, Visual Processing, and Digital Display IV, February, 1993.
  
  
  • AUTHORS: Arthur Webster, Coleen Jones, Margaret Pinson, Stephen Voran, Stephen Wolf
    
    
  • ABSTRACT: Describes an objective video quality assessment system that emulates human perception by computing scores that agree closely with quality judgments made by a large panel of viewers. The perception-based system was developed and tested for a broad range of scenes and video technologies. The 36 test scenes contained widely varying amounts of spatial and temporal information. The 27 impairments included digital video compression systems operating at line rates from 56 kbs/sec to 45 Mbs/sec with controlled error rates, NTSC encode/ decode cycles, VHS and S-VHS record/play cycles, and VHF transmission. Subjective viewer ratings of the video quality were gathered in the ITS subjective viewing laboratory that conforms to CCIR Recommendation 500-3. Objective measures of video quality were extracted from the digitally sampled video. These objective measurements are designed to quantify the spatial and temporal distortions perceived by the viewer. This paper presents the following: a detailed description of several of the best objective measurements, a perception-based model that predicts subjective ratings from these objective measurements, and a demonstration of the correlation between the model's predictions and viewer panel ratings. A personal computer-based system is being developed that will implement these objective video quality measurements in real time.
    
    
  • KEYWORDS: objective, subjective, correlation, video, quality, parameters, models, spatial, temporal, digital, compression, performance, assessment, conferencing, bit, errors, real-time
  
  
• Preliminary Results of One-Way Video Delay Measurement Algorithms, ANSI contribution number T1A1.5/92-139, July 13, 1992.
  
  
  • AUTHORS: Coleen Jones, Stephen Wolf, Margaret Pinson
    
    
  • ABSTRACT: Out-of-service measurements can be used to obtain a baseline video system delay. However, in many digital video systems, video delay is dynamic and dependent upon the scene content. This contribution describes an objective measure of video delay that can be made "in-service" and can thus track dynamic changes in the one-way video delay of a transmission channel. The metric correlates motion in the input and output video scene. A block diagram of a potential, real-time implementation is also given.
    
    
  • KEYWORDS: one-way, video, delay, objective, measure, metric, parameter, objective, quality, motion, alignment, real-time, in-service
  
  
• An Automated Technique for Measuring Transmitted Frame Rate (TFR) and Average Frame Rate (AFR), ANSI contribution number T1A1.5/92-138, July 13, 1992.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: The motion difference images of the original and degraded video sequences is used to dynamically measure the Transmitted Frame Rate (TFR) and the Average Frame Rate (AFR) of a digital video system codec (coder-decoder). The TFR measurement algorithm gives the TFR spectrum, so that in cases where the TFR is adaptive, each TFR being used by the codec is obtained. The AFR measurement algorithm gives the average number of frames per second that were transmitted by the codec during a selected time interval. The TFR and AFR measurements presented here can be used for in-service as well as out-of-service testing of digital video systems.
    
    
  • KEYWORDS: video, quality, objective, metrics, compression, frame, rate, Fourier, frequency, spectrum, performance, testing, motion, in-service, measurement
  
  
• The effect of Multiple Scenes on Objective Video Quality Assessment, ANSI contribution number T1A1.5/92-136, July 13, 1992.
  
  
  • AUTHORS: Stephen Voran
    
    
  • ABSTRACT: Presents objective to subjective correlation results when the scores from multiple video test scenes are averaged. It is shown that the prediction errors of the objective video quality model are reduced, yielding objective measurement results that more closely approximate the subjective viewing assessments.
    
    
  • KEYWORDS: video, quality, objective, subjective, metrics, correlation, scene, averaging, digital, compression, composite, system, score
  
  
• An Update on ITS Video Quality Parameters, ANSI contribution number T1A1.5/92-135, July 13, 1995.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: Presents a summary of efforts to make the objective video quality metrics more efficient in terms of computation and transmission bandwidth. Objective to subjective correlation results are presented for the subjectively rated data set given in the 1992 International Broadcasting Convention paper.
    
    
  • KEYWORDS: objective, video, quality, models, subjective, correlation, digital, compression, performance, metrics, spatial, temporal, efficient, computation
  
  
• The Development and Evaluation of an Objective Video Quality Assessment System that Emulates Human Viewing Panels, International Broadcasting Convention (IBC), July, 1992.
  
  
  • AUTHORS: Stephen Voran, Stephen Wolf
    
    
  • ABSTRACT: Discusses the approach used and the research conducted to develop an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgments made by a large panel of viewers for the subjectively rated video data set that was examined. This data set included 36 test scenes with widely varying amounts of spatial and temporal information and 27 impairments including digital video compression systems operating at line rates from 56 kbs/sec to 45 Mbs/sec with controlled error rates, NTSC encode/decode cycles, VHS and S-VHS record/play cycles, and VHF transmission.
    
    
  • KEYWORDS: objective, subjective, correlation, video, quality, parameters, models, spatial, temporal, digital, compression, performance, assessment, conferencing, bit, errors
  
  
• Spatial and Temporal Information Measures for Video Quality, ANSI contribution number T1Q1.5/92-113, January 22, 1992.
  
  
  • AUTHORS: Arthur Webster, Stephen Wolf
    
    
  • ABSTRACT: Presents an overview of metrics that can be used to quantify the amount of spatial and temporal information in a video sequence. By applying the metrics to the input and output video, the amount of spatial and temporal information that is lost by the digital video transmission system can be obtained. The spatial metrics are based on the Fourier transform of the image and the temporal metrics are based on the motion difference image.
    
    
  • KEYWORDS: digital, video, quality, measures, spatial, temporal, motion, Fourier, frequency, test, scene, information, complexity, distortions, motion
  
  
• The Development of Objective Video Quality Measures that Emulate Human Perception, IEEE Global Telecommunications Conference (GLOBECOM), December, 1991.
  
  
  • AUTHORS: Stephen Voran
    
    
  • ABSTRACT: Discusses research efforts to derive objective measures of video quality that emulate human perception. The derivation of these metrics involves the following steps: (1) a set of test scenes in selected and distorted, (2) a set of candidate objective measures are extracted, (3) a panel of viewers rates the quality of the same set of test scenes, (3) a simultaneous statistical analysis of the subjective and objective data sets reveals which portion of the objective data is meaningful, and how the objective data should be combined to create an overall metric that emulates human perception. One objective metric that correlates well with subjective quality quantifies the amount of false or extra edges that have been added to the output video. There appears to be some advantages to applying the metric separately to the still and motion portions of the video.
    
    
  • KEYWORDS: objective, subjective, correlation, video, quality, parameters, models, spatial, temporal, information, digital, compression, performance, assessment, conferencing, bit, errors, Sobel, difference, motion, still, false, edges
  
  
• Correlation Between ITS Objective Measures and Subjective Video Quality: Preliminary Results on a Set of 15 Scenes, ANSI contribution number T1Q1.5/91-124, July 9, 1991.
  
  
  • AUTHORS: Stephen Voran
    
    
  • ABSTRACT: Presents preliminary results on the objective to subjective quality correlation for a set of 5 test scenes passed through three systems (NTSC, VHS, and DS1 MPEG coding). One objective metric that correlates well with subjective quality quantifies the amount of false or extra edges that have been added to the output video. There appears to be some advantages to applying the metric separately to the still and motion portions of the video.
    
    
  • KEYWORDS: objective, subjective, correlation, video, quality, parameters, models, spatial, information, digital, compression, performance, assessment, MPEG, VHS, Sobel, difference, motion, still, false, edges
  
  
• Motion-Still Segmentation Algorithm for VTC/VT Objective Quality Assessment, ANSI contribution number T1Q1.5/91-110, January 22, 1991.
  
  
  • AUTHORS: Stephen Voran, Stephen Wolf
    
    
  • ABSTRACT: The ability of the human eye to resolve detail in a video scene is related to how much motion is present at the point of focus and whether or not the eye can track the motion. Thus, stationary portions of the video scene can be resolved in great detail by the eye, while moving portions of the video scene are normally resolved in less detail (provided the eye cannot fully track the motion). Low bit-rate digital video channels (e.g., video teleconferencing) determine how many bits are used for each local area of the video scene. Since the time averaged information content of a still video scene is much less than the time averaged information content of a moving scene, typical video teleconferencing channels can have very different static and dynamic responses. The dynamic response of the channel is also a function of the video scene and can vary on a frame-by-frame basis. Thus, it is desirable to have a general algorithm (applicable to any test waveform or test scene) that can separate the dynamic response from the static response on a frame by frame basis. This contribution describes one such algorithm. The motion-still segmentation algorithm presented here can be applied to any test waveform or test scene in order to separate the moving portions from the still portions of the video scene. A detailed description of the motion-still segmentation algorithm and its theoretical basis is given first. Then, a typical application of the algorithm is presented: measuring the increased spatial blurring of moving objects in an actual video teleconferencing scene.
    
    
  • KEYWORDS: video, teleconferencing, telephony, objective, subjective, quality, spatial, resolution, static, dynamic, motion, still, segmentation, Sobel, erode, dilate, region, growing, noise, threshold
  
  
• Features for Automated Quality Assessment of Digitally Transmitted Video, NTIA Report 264, June, 1990.
  
  
  • AUTHORS: Stephen Wolf
    
    
  • ABSTRACT: This report describes an automated method of video quality assessment based on extraction and classification of features from sampled input and output video. The first subsystem of the automated video quality measurement system is the feature extraction subsystem. Features are extracted from the sampled video that quantify many of the distortions present in modern digital compression and transmission systems. The feature measurements may then be injected into a quality classification subsystem which will determine the overall quality rating of the video. This report discusses the first subsystem of the automated video quality assessment system, namely the feature extraction subsystem. The measurement techniques used to extract a number of useful features are discussed in detail. Results are presented using sampled video teleconferencing data that contained common video compression artifacts.
    
    
  • KEYWORDS: objective, video, quality, parameters, feature, extraction, image, processing, teleconferencing, digital, impairments, blurring, blocking, smearing, jerkiness, Sobel, alignment, delay