Fast CU size and prediction mode decision algorithm for HEVC based on direction variance

As the latest video coding standard, High Efficiency Video Coding (HEVC) achieves high coding performance, up to 50% bit rate savings compared with the previous standard, H.264/Advanced Video Coding while maintaining the same output video quality. Intra-coding in the HEVC significantly improves the compression efficiency, due to the various kinds of coding unit (CU) sizes, and high density of angular prediction modes. However, the improvement of the coding performance is obtained at the expense of the extraordinary computation complexity, which obstructs the usefulness of HEVC encoder for real-time applications. This paper presents a novel intra-encoding algorithm for HEVC, which is composed of an efficient CU partitioning technique and a fast intra-mode decision method based on direction variance. We utilize a preprocessing method in which the texture complexity and direction energy distribution along a special direction are extracted from a macroblock. According to the texture complexity obtained, an early CU splitting termination is proposed to decide whether a CU should be decomposed into four lower-dimensions CUs or not. Furthermore, a subset of prediction modes in accordance with the dominant direction energy distribution is chosen for the further rough mode decision and rate–distortion optimization process. The simulation results indicate the algorithm based on image understanding achieves better trade-off between rate–distortion performance and complexity reduction than the previous algorithms. Compared with the reference software HM16.7, the proposed algorithm can save 57% coding time on average, with a negligible bit rate increase of 0.65%, and quality losses lower 0.08 dB, respectively.