The traditional craft of cutting diagrams as a front-end is a task that most front-end developers are unwilling to face.

In order to solve the various problems of cutting diagrams, people have racked their brains to develop a variety of design draft-to-code (D2C) tools, and these D2C tools continue to iterate as the software used by designers changes.

From the era of Photoshop, the front-end needs to manually mark nodes for separate style export (as shown in Figure 1), to sketch measure, which can be output as a whole page (as shown in Figure 2), and its efficiency and results have been qualitatively improved.

However, the problem of cutting pictures has not been completely solved, because the information contained in the design draft is only responsible for outputting the style, and there is no way to output the layout of the web page. We still have no way to directly copy the generated code and use it directly in our project.

figure 1

figure 2

In the process of studying the existing D2C cases, we found that many mature solutions reference the generation of machine learning-assisted code, most of which are used for visual recognition and semantic recognition, so we wondered whether machine learning can Applied to the layout of the web page?

In order to verify our conjecture and solve the actual pain points in our work, we decided to develop a D2C tool ourselves, hoping to help us realize the process of web page layout through the machine. The overall workflow is roughly as shown in Figure 3. First, get the metadata of the design draft data, and then perform a series of processing on the data of the design draft to export the free dsl, and then generate the code of the corresponding end according to the dsl.

image 3

Linear layout

Flow layout

grid layout

According to the usual habit of cutting graphs, we will first identify whether there is an obvious up-down or left-right positional relationship between a group of level nodes, and then put them into the grid, and finally the nodes that are independent of these nodes are absolutely positioned. .

Letting the machine identify the positional relationship between nodes has become a key part of solving the problem.

At first, we hoped to judge the layout by writing certain rules, and make the layout by judging whether the positional relationship between the front and rear nodes is up and down or left and right. However, the rules that only focus on the positional relationship of the two nodes are difficult to judge the absolutely positioned nodes. , and fixed rules are always inflexible. So we thought of machine learning, which is good at dealing with classification problems.

Obviously, a machine learning model trained with a large amount of data can well imitate our usual habit of cutting pictures, and can be more flexible when dealing with various edge scenarios. As long as a reasonable model design is carried out, it can assist us in the layout processing.

For a specific node, we have no way to judge its real layout. Only by putting it in the document flow and looking at the context can it reflect its actual meaning.

In dealing with part-of-speech tags, RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory Network) are more suitable for such scenarios. As a classic neural network model, RNN is established by bringing the weights of the previous training into the next training. Contextual association, LSTM as a variant of RNN solves the long-term dependency problem that RNN is difficult to solve, and it seems to be a good choice for training web page layout.

RNN (Recurrent Neural Network)

665px LSTM (Long Short Term Memory Network)

Using LSTM can indeed solve our problem, but due to the dependence of such neural networks on time series, there is no way to perform parallel operations on the context data, which makes our computer unable to train the model more efficiently, and the web page layout only needs to obtain different nodes There is no strong requirement for the loading order.

With the publication of an article “Attention is All you Need” in 2017, the whole field of machine learning has ushered in a new round of revolution. The most mainstream frameworks transformer, BERT, and GPT are all based on attention. developed.

Self-attention The self-attention mechanism is a variant of the attention mechanism. The weight information of a single vector in the global is obtained through the global association weight. Because each node adopts the same operation method, the nodes in the same sequence can be processed at the same time. Context computing greatly improves the efficiency of model training and makes it easier for us to optimize and regression models.

self-attention

(1) After obtaining a set of data, in order to remove the influence of the size of the value, we first normalize the data once, and divide each value of the input data by the maximum value in the set of data.

(2) Perform three linear transformations on each set of data to obtain the corresponding Q, K, and V of each set of data, and then we can perform the self-attention operation.

(3) Let’s take node1 as an example. If you need to calculate the correlation between node1 and other nodes, you need to use Q1 to perform a dot product operation with the key of each node, and multiply their sum with V1. After the multiplication is too large, the last softmax calculation is performed to obtain the context information of node1 and other nodes.

As a supplement to the markup design draft, the real data in the web page is also a reliable data source, but the biggest difficulty in the process of crawling the web page is to determine whether the nodes in the page are horizontal or vertical.

Due to the strange ways to achieve horizontal arrangement, we can use float, inline-block, flex and other methods. If we only get the positioning and width and height information of the nodes in the web page, we still need to manually mark his layout, so we still need to start from the css of the nodes. Start with manual filtering after batch acquisition to remove low-quality data. It is an effective complement to the marked design draft.

Machine learning is very good at dealing with classification problems. Compared with the traditional handwritten rule layout, machine learning is trained based on our existing development habits, and the final generated code is also closer to our usual habit of cutting pictures. The readability of the code and Maintainability is even better.

Compared with static pages handwritten by different people, machine-generated static pages follow consistent code specifications, and the code style is more uniform.

In the process of model building, specific usage scenarios can be compared to the content in the field of text or images, which is convenient for finding existing models for transfer learning.

The core of using machine learning to solve the problem of web page layout is to establish the contextual association of nodes. By understanding the operating principles of various classic neural network models, we have selected the recurrent neural network and self-attention mechanism, which can establish contextual association models. Through further understanding of its operating principle, we have chosen a self-attention model that is closer to the web page layout scenario and has higher operating efficiency. It can also be seen that understanding the operating mechanism of the model can better help us solve practical application scenarios. .