Author | Tornado

Guided reading: With the development of the computer vision field, animation has become more and more cool. When improving image quality, various motion solutions also need to take into account file size and performance. The transparent MP4 solution enables the designer to make animation WYSIWYG, which fully utilizes the designer’s imagination and creativity. Baidu YYEVA dynamic effect player is a set of lightweight, high-performance, cross-platform dynamic effect solutions based on transparent MP4. It supports the insertion of dynamic elements, provides a complete tool chain, and provides a one-stop solution from the design side. resource export, online preview, and client-side rendering SDK.

The full text is 3736 words, and the estimated reading time is 10 minutes.

01 Introduction of YYEVA

YYEVA has implemented a complete set of tool chain, including: AE plug-in at the resource output end, online preview tool, client-side rendering SDK, transparent MP4 resources, and dynamic business elements can be inserted.

• YYEVA is a lightweight, high-performance, cross-platform, dynamic MP4 resource solution

• YYEVA includes a complete tool chain from design tools AE plug-ins, online preview tools, and client-side rendering SDKs

• Based on the affine matrix operation, the position information of each frame of the layer is obtained

• Integrated MP4 (avc/hevc) encapsulation protocol and other related functions

• Highly extensible, can restore all the details of the designer

• Support Web, Android, iOS, etc.

YYEVA has been used in many projects, among which YYEVA, Tieba, Baidu, Kankan and other scenes use YYEVA to achieve complex dynamic effects, and are also provided to the project team of external companies for use

Click the link to view the case demo video: https://mp.weixin.qq.com/s/bXgauBqtwUlT8IRXPLr45w

Open source project address: https://github.com/yylive/yyeva

YYEVA official website: https://yyeva.netlify.app/

02 YYEVA’s Path of Exploration

2.1 Centralized implementation of animation

1. Results-Oriented Recording

This method is to record the image of each frame of the animation, and restore the animation effect according to the RGBA of the image during playback.And only the final result is recorded, the animation elements of the design cannot be restored, so it is difficult to modify the animation elements, and inserting elements is also more complicated

WYSIWYG can restore all design effects; no need to develop support for specific special effects; the number of elements and motion complexity have little impact on playback performance

Large file size; not easy to support dynamic insertion or replacement of elements

2. Process Oriented Recording

This method is to record the animation creation process, and at the playback end, calculate the motion trajectory of each element according to the process, and restore the effect; real-time calculation is required to restore the animation, and the more complex animation, the greater the amount of calculation, such as the calculation related to filters and Bezier curves Very performance-intensive.

Small in size, can be stretched at will without affecting the quality; easy to dynamically insert elements

Poor performance, consumes CPU and GPU; poor support for complex animations, the more complex animations, the easier it is to freeze

Comparing the realization principles of the above two animations, in order to achieve the WYSIWYG effect and give full play to the designer’s imagination, YYEVA adopts a result-oriented animation method—transparent MP4 scheme

Compared with the sequential frame scheme, transparent MP4 has the advantage of higher compression rate, thus solving the problem of large file size.We have developed a YYEVA toolchain that supports dynamic insertion or replacement of elements

2.2 Video animation

With H264 encoding, the color sampling standard of MPEG-4 is YUV, YUV is the superposition of luminance and chrominance components, and does not support alpha channel. Therefore, how to make MP4 video support transparency, the common way in the industry is to use two channels to store separately RGB data and Alpha data of the video. Due to the advantages of video animation WYSIWYG and support for more complex special effects, it is currently widely used in various scenes of YY, and has become the preferred solution for YY animation playback.

If the animation resolution is 500×500, the resolution of Mp4 is 1000×500, of which 500×500 on the left is RGB data, and 500×500 on the right is Alpha data. Player is combined into RGBA texture and then rendered and displayed.

2.3 Mixing MP4 motion

In MP4 motion effects, add some business elements such as nicknames, avatars, pictures, etc. The common practice is to superimpose a native View when MP4 is playing, or use MP4 + SVGA/Y2A to achieve. In this way, there are often problems such as resource download, synchronous playback, and maintenance of multiple sets of resources.

2.4 YYEVA solution

The video frame and description information are combined into one MP4 resource and rendered synchronously, which solves the problem of resource download, synchronous playback, and maintenance of multiple sets of resources.

The Json description information of YYEVA is as follows:

1. YYEVA describes dynamic elements

Describe several major elements of an animation: time, position, deformation

• Time: Describe the frame index that needs to appear through the frameIndex of Json

• Position: Describe the position and size of the element on the canvas through Json’s RenderFrame

• Deformation: Get the mask of element deformation through Json’s OutputFrame

2. Demonstration of mask mask

Shapes can be recorded in 2 ways

03 YYEVA implementation plan

The frame diagram of YYEVA is as follows:

The flowchart of the toolchain is as follows:

3.1 YYEVA plug-in

The YYEVA solution analyzes the relevant layer information made by the designer according to the specification through an extension program developed on AE, and exports a YYEVA resource through the layer analysis module, h264 module, and resource synthesis module of the YYConveterMP4 plug-in.

1. Layer parsing module

• Normative testing

• Whether the selected composition contains transparent area layers: The resources currently processed by the plugin are based on YY transparent MP4 resources, so the source material must be a transparent MP4 resource.That is, RGB/Alpha is separated from left to right

• Whether the selected composition contains the Mask mask area: the text mask area uses mask_text as the composite name; the image mask area uses mask_image as the composite name

• Whether to include the YYConverterMP4 template, which is used by the following H264 module to convert MP4

• Layer handling

• Calculate affine matrix: Matrix

• Calculate Mask Position: RenderFrame

• Extract Mask Compositing

• Calculate the RenderFrame of each frame of all layers under the Mask composition

• Copy an output composition, and adjust the position after reducing the Alpha area by 0.5 times. At the same time, it is divided into three parts: rgb, alpha, mask

• Copy all valid mask layers to the output composition, adjust to the mask area, and calculate the OutputFrame

• Resize the area of ​​the output composition

• Combine the RenderFrame and OutputFrame calculated above into one Json data

• copy synthesis

After making a copy of the formulated output composition, adjust the position after reducing the alpha area by 0.5 times, and divide it into three parts at the same time: rgb, alpha, and mask.

2. H264 module

• Create a specified YYConverterMP4 template on AE, and output the unified rendering queue into avi format

• Integrate the ffmpeg command line tool inside the plug-in, and invoke the command line tool through child_process. The ffmpeg executable files used by MAC/Windows are different (.app and .exe files)

• When using ffmpeg to convert, unify the output image quality and volume by encapsulating the CRF parameters of three gears

• It is realized by selecting different encoders to output H264/H265 resources

3. Resource synthesis module

• Packed data
  According to the basic information of outputFrame, inputFrame, and video, form a json

• data compression coding

  Package and encode json into base64, and add the prefix and suffix yyeffectmp4json[[ Json ]]yyeffectmp4json

• export to AVI

  Use the created template YYConverterMP4, add the copied composition to the rendering queue, and output avi resources

• output to MP4

  Use the H264 module to encode into h264/h265 resources

• Write data to MP4

  Write the json data to the metadata section of the h264/h265 resource using ffmpeg

3.2 YYEVA client rendering

The overall architecture of YYEVA rendering SDK is shown in the following figure:

After the client reads a YYEVA resource, it will go through the following rendering process:

• Use the resource parsing module to parse the Json data integrated in the Metadata segment

• Then through the element parsing module, the dynamic Json data is modalized into corresponding objects.

• Through the audio and video module, decode the audio and video tracks of MP4 resources

• Render the video track + Mask dynamic elements to the screen one by one through the rendering module

The entire rendering process is as follows:

1. Extract the description information

There are two ways to extract the Metadata information of MP4 as shown in the figure

When the plugin writes json data to MP4, we wrap the data in a layer of yyeffectmp4json[[jsondata]]yyeffectmp4json, this operation is mainly to quickly extract jsondata according to character matching

2. Implementation logic of client-side rendering

3. RGB+Alpha mixing

4. Mask mixing

5. Rendering diagram

04 Conclusion

YYEVA is currently open source, including AE plug-ins, client-side rendering SDK (web/iOS/android), and friends who have other needs and ideas are welcome to leave a message in the comment area or submit valuable issues directly to the code repository.

Welcome everyone to star, your attention is our biggest motivation.

In the era of the current live broadcast business, dynamic playback solutions are also constantly developing and progressing, and various new dynamic effect solutions are emerging one after another. It is foreseeable that we will continue to explore and optimize in the future. Dynamic element rotation, support for more AE features, combined with AI technology, provide users with a better experience.

————END————

Recommended reading:

A brief analysis of the asset system architecture of Baidu’s trading platform

Understand APP speed test from zero to one

Flow logs can easily cope with complex scenarios of “1 billion IP pairs” and realize network traffic visualization of ultra-large-scale hybrid clouds

Baidu App Android Startup Performance Optimization – Tools

Application of digital human technology in live broadcast scenarios

Baidu engineers teach you to play with design mode (factory mode)

Super model engineering practice polishing, Baidu Intelligent Cloud releases cloud native AI 2.0 solution

Efficiency improvement practice of front-end and back-end data interface collaboration