Image Quality Assessment

Personnel (Past, Present): Tamir Hegazy, Dogancan Temel, Mohit Prabhushankar

Goal: The goal in the field of Image Quality Assessment (IQA) is to objectively estimate the subjective quality of an image. Datasets are obtained by distorting pristine images and asking multiple humans to rate the quality of the distorted images. Algorithms are designed to estimate this quality.

Challenges: Estimating the visual quality of an image requires a machine vision algorithm to understand the constraints and nature of human visual systems and apply them to estimate quality. However, since the nature of the visual system is not completely understood, IQA is a challenging task. Adding to this challenge is the subjective nature of human judgment. Finally, incorporating known visual system features like sparsity, Just Noticeable Difference, Edge-color tradeoff among others in a machine learning pipeline is a further challenge.

Our Work: The challenges presented by IQA are ideal in studying the congruence (or incongruence) between human and machine vision. At OLIVES, we study IQA from a data-driven and an image and signal processing perspective.

We identify the effect of distortions in the spectral domain in [1]. This analysis of the spectral domain is extended to multi-channel analysis in BleSS [2] and a multi-scale and multi-channel analysis in SUMMER [3]. We study the incongruence between pristine and distorted images based on their color and structure in CSV [4]. Analysis of the effect of distortion on keypoint detection is proposed and performed in the RESIFT IQA metric [5].

The challenge to solve in data-driven algorithms is to incorporate known facts and constraints about the visual system in machine learning algorithms. We use the concepts of sparsity, color-intensity channel analysis, and JND-based feature rejection to construct an unsupervised IQA algorithm called UNIQUE [6]. UNIQUE is extended by analyzing the network itself to weigh the edges and color components differently in MS-UNIQUE [7]. Both UNIQUE and MS-UNIQUE are combined in [8]. We provide a unique perspective to evaluation of IQA algorithms where we show distortion level-wise evaluation in [9].

We dig deeper and propose a new concept of using gradients to characterize distortions in neural networks [10]. Specifically, we posit that neural networks are trained to ignore distortions if trained on pristine data and suggest that gradients provide directional information regarding presence of distortions in manifolds. This paper won the Best Paper Award at ICIP 2020. We further analyze gradients and view them as answers to logical contrastive questions of the form `Why P, rather than Q?’ and explain IQA algorithms in [11]. Finally, we analyze gradients in IQA from the perspective of Free-energy principle in [12].

Extensions of IQA works in other applications: The generality of studying machine and human vision allows application of this work in multiple applications. We propose the concept of Extended ZCA whitening to extend MS-UNIQUE for texture retrieval in [8]. The architectures of UNIQUE and MS-UNIQUE are used for seismic analysis in [13] and [14]. Specifically in [13], the uncertainty principle between orthogonal domains is utilized to detect salt domes and in [14], the features learned from natural images are transferred over to seismic images to create generalizable interpretations of salt domes, horizons, seismic reflections and faults in the F3 Block, SEAM and Netherlands datasets. All this is done in an unsupervised fashion.

Awards and Highlights: Our work on IQA won the Best Paper Award at ICIP 2019 (Top 1 among 1000 accepted papers) and an IEEE SPS Travel Grant Award to Taiwan. Our work on contrastive explanations on IQA won the Most Viewed Special Session Paper Award at the virtual ICIP 2020 conference.

References:

  1. T. Hegazy and G. AlRegib, “A New Full-Reference IQA Index Using Error Spectrum Chaos,” in IEEE Global Conference on Signal and Information Processing (GlobalSIP), Atlanta, GA, Dec. 3-5 2014. [PDF][Code]
  2. D. Temel and G. AlRegib, “BLeSS: Bio-Inspired Low-Level Spatiochromatic Similarity Assisted Image Quality Assessment,” in IEEE International Conference on Multimedia and Expo (ICME), Jul. 11-15 2016. [PDF][Code]
  3. D. Temel and G. AlRegib, “Perceptual Image Quality Assessment Through Spectral Analysis of Error Representations,” in Signal Processing: Image Communication, vol. 70, pp. 37-46, 2019. [PDF][Code]
  4. D. Temel and G. AlRegib, “CSV: Image Quality Assessment Based on Color, Structure and Visual System,” in Signal Processing: Image Communication, vol. 48, pp. 92-103, Oct. 2016. [PDF][Code]
  5. D. Temel and G. AlRegib, “ReSIFT: Reliability-Weighted SIFT-based Image Quality Assessment,” in IEEE International Conference on Image Processing (ICIP), Sep. 25-28 2016. [PDF][Code]
  6. D. Temel, M. Prabhushankar, and G. AlRegib, “UNIQUE: Unsupervised Image Quality Estimation,” in IEEE Signal Processing Letters , vol. 23, no. 10, pp. 1414-1418, Oct. 2016. [PDF][Code][Link]
  7. M. Prabhushankar, D. Temel, and G. AlRegib, “MS-UNIQUE: Multi-Model and Sharpness-Weighted Unsupervised Image Quality Estimation,” in Image Quality and System Performance XIV, part of IS&T Electronic Imaging, San Francisco, CA, Jan. 29 2017. [PDF][Code]
  8. M.Prabhushankar, D. Temel, and G.AlRegib, “Generating Adaptive and Robust Filter Sets Using an Unsupervised Learning Framework,” in IEEE International Conference on Image Processing (ICIP), Beijing, China, Sep. 12-20 2017. [PDF][Code]
  9. M. Prabhushankar*, G. Kwon*, D. Temel, and G. AlRegib, “Semantically Interpretable and Controllable Filter Sets,” in IEEE International Conference on Image Processing (ICIP), Athens, Greece, Oct. 7-10 2018 [PDF][Code]
  10. G. Kwon*, M. Prabhushankar*, D. Temel, and G. AlRegib, “Distorted Representation Space Characterization Through Backpropagated Gradients,” in IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, Sep. 2019. [PDF][Code]
  11. M. Prabhushankar, G. Kwon, D. Temel, and G. AlRegib, “Contrastive Explanations in Neural Networks,” in IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, Oct. 2020. [PDF][Code][Video]
  12. M. Prabhushankar and G. AlRegib, “Stochastic Surprisal: An Inferential Measurement of Free Energy in Neural Networks,” Frontiers in Neuroscience – Perception Science, submitted on Apr. 22 2022.
  13. M. A. Shafiq, M. Prabhushankar, and G. AlRegib, “Leveraging Sparse Features Learned From Natural Images for Seismic Understanding,” in EAGE Annual Conference & Exhibition, Copenhagen, Denmark, Jun. 11-14 2018. [PDF][Code]
  14. M. A. Shafiq, M. Prabhushankar, H. Di, and G. AlRegib, “Towards Understanding Common Features Between Natural and Seismic Images,” in Expanded Abstracts of the SEG Annual Meeting, Anaheim, CA, Oct. 14-19 2018. [PDF]

Print Friendly, PDF & Email