The key idea of the paper is view-substitution technique.
They use volume rendering in refernce view with target view color and obtain residual between the rendering in refernce view with reference view color.
The intuition is that the residual must not have high-frequency since the View Dependent Effect (called VDEs) is applied similarly in all image pixels. To enforce this, they utilizes bilateral solver.