Compressive Sensing and Precomputed Radiance Transfer?

Following the development in Peers et. al.’s Compressive Light Transport Sensing (download), the light transport problem can be described by the following matrix equation:

$\mathbf{c} = T^T \mathbf{l}$

where

$\mathbf{c}$ is a $p$ -dimensional vector, where $p$ is the number of pixels in the final image, and each component of $\mathbf{c}$ is the color of a pixel,
$\mathbf{l}$ is an $n$ -dimensional vector, where $n$ is the number of light sources, and each component of $\mathbf{l}$ is the intensity of a light source, and
$\mathbf{T}$ is a transfer matrix of size $n \times p$ .

The $i$ -th column of $T$ , which we shall denode by $\mathbf{t}_i$ , allows us to write the color $c_i$ of Pixel $i$ as $c_i = \mathbf{t}_i^T \mathbf{l}$ .

Each $\mathbf{t}_i$ is a signal the paper wishes to learn from real measurements. A measurement consists of shining a predetermined pattern of light to the scene and take a photograph. So let’s say we make $m$ measurements. Now, the light vector $\mathbf{l}$ becomes a matrix $L$ of size $n \times m$ , the color vector $C$ of size $p \times m$ , and the equation changes to:

$C = T^T L$

or, if you prefer to write the signal we want to reconstruct at the end,

$C^T = L^T T$ .

Notice that the above equation gives rise to $p$ indepedent signal reconstruction problems for each column where $L^T$ acts like a measurement ensemble.

Suppose that each $\mathbf{T}_i$ is sparse in some basis $B$ , having at most $\mathbf{S}$ non-zero terms. We have that

$C^T = L^T T = L^T B B^T T = (L^T B) \hat{T}$ .

where $\hat{T}$ is $T$ projected into $B$ .

The theory of compressive sensing states that, if $L^T B$ satisfies the restricted isometry property (read more about this property here) and

$m \geq \alpha S \log n$ .

some constant $\alpha$ , then we construct each $\mathbf{t}_i$ exactly or with minimal error.

Application to Precomputed Radiance Transfer?

Can we apply the above compressive sensing framework to PRT? After all, the core tasks of image relighting and PRT are the same: learning the transfer function of each pixel. The difference is that measurements in PRT are done in computer by tracing rays.

It’s possible. In the scenario that you would like to relight a scene by a high resolution environmental map, say $256 \times 256 \times 6$ (as done in this paper). The transfer vector $\mathbf{t}_i$ for each pixel would also be that high in dimension. To enable real time relighting, we wavelet-transform these transfer functions and keep only about 200 largest coefficients. Here, $\log_2 n$ is about 20. So, we can render about 4000 images and use it to reconstruct the functions.

However, all of the above is not so impressive. First, it might take a very long time to render that 4000 images accurately. Second, we only assume that each pixel has one color, which either mean that the view is fixed or all the material is diffuse. Again, I don’t know how to handle variable views yet, but surveying more literature might help.