A Semi-automatic Oriental Ink Painting Framework for Robotic Drawing from 3D Models

The problem of extracting and drawing expressive lines from 3D models is equivalent to the problem of how artists create line drawings from real objects, which is one of the challenging task in NPR. The detailed survey of line drawing methods from 3D models can be found in Bénard and Hertzmann’s tutorial [10]. Saito and Takahashi [11] first created stylized contour lines and curved hatching from 3D models by using 2D image processing operations. Winkenbach and Salesin [12] extended the work of [11] to generate more realistic line drawings from 3D models by integrating 2D and 3D rendering. Zhang et al. [13] presented a cellular automaton model to simulate the Suibokuga-like painting of 3D trees. DeCarlo et al. [14] proposed Suggestive Contours to improve the quality of line drawings by connecting perception knowledge and differential geometry. Grabli et al. [5] created vivid stylized line drawings such as Japanese big brush from 3D models based on the work of [14]. Judd et al. [15] defined view-dependent curvature to generate Apparent Ridges that captured more detailed features than the Suggestive Contours. Recently, Uchida and Saito [16] introduced two fully convolutional neural networks to determine the intensity of strokes for stylized line drawings. Liu et al. [6] generated impressive stylized strokes for line drawings based on a combination of a differentiable geometric module and an image translation network. Although these methods have ability to extract feature lines and generate visually pleasing digital strokes from 3D models, they are unsuitable for creating physical stylized strokes for robotic drawing because the extracted lines are likely to be redundant or oversimplified, and the styles of digital strokes which are usually implemented by texture mapping are difficult to be realized in physical drawing. Inspired by the work of interactive inking for cleaning rough sketches [17], our work does not pursue fully automatic realization of line drawings but focuses on extracting a few non-overlapping expressive lines with a combination of existing automatic contour extraction method and human-computer interaction process.

The earliest work of robotic drawing could date back to Harold Cohen’s exhibitions at the Computer Museum in 1995 [18]. Cohen developed a robotic drawing system “AARON” that could color its own paintings using a variety of brushes. Thanks to advances of sensor and AI technologies, the diversity, perception and collaboration skills of robots improved significantly. Calinon et al. [19] designed a $4$ Degrees of Freedom (DOFs) robotic arm to draw a portrait detected from a webcam using traditional image processing and inverse kinematics methods. Löw et al. [9] designed a robotic system “drozBot” to draw artistic portraits from images based on a novel ergodic control algorithm. However, the above-mentioned methods can only create monochrome paintings. Recently, Lindemeier et al. [7] proposed a painting robot to realize colored oil painting styles from 2D images. Luo and Liu [20] realized an interesting colored Cartoon style portrait painting using NPR techniques. Scalera et al. Karimov et al. [21] developed a data-driven model for accurately color mixing and reproduced famous artworks based on a robotic arm. More recently, the deep learning methods have been developed to solve more complicated robotic drawing problems. Zhang et al. [22] trained a network to identify the type of individual strokes for intelligent calligraphy beautification. Gao et al. [23] developed a robotic system for efficiently drawing portraits from images based on a combination network of Neural Style Transfer and Generative Adversarial Network. Bidgoli et al. [24] used deep learning method to learn stylized brushstroke from human artists and reproduce them through robotic painting. Furthermore, the surface of robotic drawing has been extended from 2D to 3D. Song et al. [25] presented an impedance control method that can draw user’s 2D drawing on a 3D surface without vision support, and extended the drawing on a large and nonplanar surface with a mobile platform [26]. In contrast, Liu et al. [27] developed a robotic system to draw 2D strokes on 3D surface based on scanned point clouds and a robust motion planning algorithm. While all these robotic drawing methods have varying focuses such as artistic portrait drawing [19, 20, 23, 9] or stylized colored painting from 2D images [7, 20, 8, 28] or realistic calligraphy drawing [22] or robust drawing on 3D surfaces [25, 26, 27], they haven’t considered creating stylized ink paintings from 3D models which requires understanding the abstraction technique from 3D models and the motion control of a robotic arm and a brush to generate the styles of oriental ink painting such as Noutan and Kasure.

Inspired by the investigation that digital line drawings can depict 3D model that even match the artist’s drawings [30], we select three typical geometry-based digital contours, namely occluding contours (OCs), suggestive contours (SCs)[14], and apparent ridges (ARs) [15], as the basis for the creation of ink paintings from 3D models. Given a camera viewpoint and a 3D model, the combination of these contours produces a raster contour image as shown in Fig. 3 (a-d), where OCs conveys the rough shape of the object, SCs and ARs add considerable amount of details to the 3D model.

Although the extracted contours are a reasonable description of 3D shape, it creates large amount of dense and detailed lines that are inapplicable for oriental ink paintings which focus on sparse stylized strokes. Motivated by the CNN-based neural network for sketch simplification [31], we employ the similar network with 3 down-convolution layers, 17 flat-convolution layers and 3 up-convolution layers to simplify the contour image. The loss function is defined as the $l^{2}$-norm of the difference between the output image and the target image. However, the dataset of the previous network is for cleaning rough pencil sketches. Therefore, we construct our contour simplification dataset through manual annotation.

The dataset consists of two parts, one part is rough contours automatically extracted from 3D models and the other part is the corresponding feature lines selected by users. The 3D models in the dataset are constructed from $4$ categories including humans, flowers, animals and furniture, and each category has $20$ 3D model as shown in Fig. 4. To make the training robust, we augmented the dataset by applying image blurring, flipping and adding noise. Then, we measured precision by defining the contour pixels as “True” and the rest pixels as “False”. The dataset was divided into training set, validation set and test set with a ratio of $8:1:1$. After $600$ epochs of iterative training with batch size of $8$, the precision of the network converged to $0.982$ and $0.853$ on the training set and the validation set when considering only one viewpoint, and the corresponding accuracy converged to $0.986$ and $0.857$ when considering two viewpoints as shown in Fig. 5, which indicated that increasing the number of viewpoints did not change the network accuracy much.

To generate smooth individual curves without junctions for robotic drawing, locating proper corners is critical because corners have significant variations in curvature. Thus, we use a robust corner detector to remove junctions or intersection points at which curves meet based on local and global curvatures [32]. The local curvature of the $i$th contour pixel $(x_{i},y_{i})$ in a raster image is denoted by $\kappa_{i}$. The pixels with the maxima of absolute curvature $|\kappa_{i}|$ are selected as corner candidates. However, round corners which are important for forming smooth curves might be incorrectly marked as candidates. Therefore, a global curvature threshold $\overline{\kappa_{i}}$ is defined to remove round corners by considering the mean curvature within a region of support $[L_{left},L_{right}]$:

where $L_{left}$ and $L_{right}$ are the nearest curvature minima on both sides of pixel $i$, and $\mu$ is the splitting ratio to control the number of vectorized curves. If $|\kappa_{i}|$ of a pixel is less than $\overline{\kappa_{i}}$, it is classified as round corner and then should be removed from the corner candidates. Finally, the rest corner pixels are utilized as endpoints $X_{k}$ to form individual polylines, and each polyline is represented by a series of line segments $\left\{{(X_{1},X_{2}),(X_{2},X_{3}),...,(X_{k},X_{k+1}),...,(X_{n-1},X_{n})}\right\}$.

To generate appropriate stylized strokes for oriental ink painting which emphasizes sparse and expressive contours, we developed a user interface (UI) as shown in Fig. 6 to allow users to flexibly edit the simplified contours. The left view of the UI is designed for human-computer interaction, and the right view simultaneously shows the completed digital ink painting. The UI consists of several tools including “Split”, “Pick”, “Delete”, “Merge” and “Insert”.

Split. A “Splitting ratio” slider is used to adjust the splitting ratio $\mu$ for vectorization. Small $\mu$ will break up long curves and generate short polylines. In contrast, large $\mu$ will retain long curves. The default value of $\mu$ is set as $160$.

Pick. The “Pick” tool allows users to interactively select expressive polylines. By default, the vectorized outside OCs are automatically selected as the target as shown in Fig. 6(a). However, the inside polylines require manual selection because not all the simplified SCs and ARs are appropriate for stylized painting. Once a polyline is selected, the users can extend the polyline by inserting new points.

Delete. The “Delete” tool is used to remove incorrectly picked polylines.

Merge. For over-simplified or over-segmented curves, the “Merge” tool can be used to re-connect broken polylines and form long curves.

Insert. The “Insert” tool is used to add new polylines. The missing features may happen during the feature extraction step or after neural simplification. To ensure the expressive features are correctly added or selected, our interface refers not only to the vectorized contours but also to the original model as shown in Fig. 6(b).

Moreover, we design two more sliders to control the maxima and minima width (thickness) of a stroke. While the interaction process is not fully automatic, users can obtain expressive strokes according to their aesthetic understandings by taking advantage of the editing tools.

The extracted polylines may not be smooth or unevenly sampled, which are not suitable for direct use in robotic drawing. A B-spline curve can generate evenly distributed smooth curve with several control points and avoid the Runge phenomenon. However, the choosing of control points is crucial because the generated curve does not necessarily pass through the sample points as shown in Fig. 7(a). Considering a cubic B-spline curve $C(t)=\sum_{i=1}^{m}N_{i,3}(t)P_{i}$, where $P_{i}$ is control point and $N_{i,3}(t)$ is the basis B-spline of degree $3$. To compute a smooth curve for approximating endpoints $X_{k}$ of extracted lines, we employ a nonlinear optimization method [33] to fit $X_{k}$ using the following object function:

where $D_{sdm}(C(t)-X_{k})$ is the squared distance error term to measure the distance between the point $X_{k}$ and the curve $C(t)$, $f_{1}=\int||C^{\prime}(t)||^{2}dt$ and $f_{2}=\int||C^{\prime\prime}(t)||^{2}dt$ are smoothness terms, and $\alpha,\beta\geq 0$ are constant coefficients. The minimization problem of the object function $f$ can be solved by the Quasi-Newton iteration method to generate an updated fitting curve as shown in Fig. 7(b-d). After 15 iterations, the updated curve almost pass through all the sample points as seen in Fig. 7(d).

The thickness of a stroke $t_{i}\left\{i=1,2,...,n\right\}$ at each of sampling vertices $C_{i}(x_{i},y_{i})$ on the optimized B-spline curve is defined by an exponential interpolation of a minimum thickness $t_{min}$ and a maximum thickness $t_{max}$ in the middle of the curve:

where $\gamma=0.6$ is an exponent to control the change rate of the stroke thickness [5]. Fig. 7(e) shows the simulated drawing stroke with thickness controlled by Eq. (3) when the default values of $t_{max}$ and $t_{min}$ were set as $25$ and $4$ respectively.

In order to physically draw strokes that are consistent to the simulated strokes as shown in Fig. 7(e), firstly, we carried out an experiment to derive the relationship between stroke thickness $t_{i}$ and the descent of brush $h_{i}$ which is controlled by a robotic arm. Let $h_{tip}$ be the height at which the brush tip just reaches the paper without distortion. The descent step of the brush was set as $2mm$, and $(h_{i}-h_{tip})$ was updated from $2mm$ to $18mm$. The footprint of the brush is like a droplet and the increasing descent results in more severe brush bending, which in turn leads to wider stroke as shown in Tab. I. The thickness of each stroke $t_{i}$ was measured at the widest part of the stroke. Then, a standard least-squares fitting approach was applied to estimate the relationship between $t_{i}$ and $h_{i}$. Finally, we obtained $h_{i}=wt_{i}+b+h_{tip}$, where $w=1.178$, $b=-0.801$, and the coefficient of determination $R^{2}$ is 0.957, which indicates that the linear fitting is appropriate for thickness prediction. Correspondingly, the stroke position $(x_{i}^{\prime},y_{i}^{\prime})$ for robotic drawing can be mapped by $(x_{i}^{\prime}=w(y_{i}-y_{c})+x_{o}^{\prime},y_{i}^{\prime}=w(x_{i}-x_{c})+y_{o}^{\prime})$, where $(x_{c},y_{c})$ is the centroid of sampling vertices in simulation space, and $(x_{o}^{\prime},y_{o}^{\prime})$ is the center of robot Cartesian space. Fig. 7(f) shows the robotic drawing stroke which is close to the simulated one after using stroke mapping.

In traditional oriental ink paintings, the patterns of light and darkness can be applied to strokes by using two typical ink painting styles, which are termed as Noutan (shade) and Kasure (scratchiness) [13, 34] as shown in Fig. 2 and Fig. 8. Noutan uses water to lighten the ink and creates shade of gray color of a stroke. Fig. 8 (a, b) shows the light gray, dark gray and gradient shading strokes drawn by different Noutan shading styles. Kasure is the scratchy break-up of a stroke trajectory caused by insufficient and uneven supply of ink along the brush as shown in Fig. 8(a). In our work, we explore an automatic approach to achieve the two painting styles using robotic arm.

To create the shading effect, we assume that Noutan is a diffusion process of a brush with varying color of ink. Therefore, two ink stones, which are filled with light ink and thick ink, are prepared for mixing the two kinds of inks using the brush as shown in Fig. 9.

In our experiment, we observed that fully moistening the brush in the thin ink and scraping the ink at the edge of ink stone in advance is helpful for forming natural Noutan style. Thus, we defined a series of actions for automatically rendering the Noutan style:

where $A_{tran}$, $A_{dip}$, and $A_{scrape}$ represent the actions of translation, dipping and scraping of the brush; $P_{light}$, $P_{thick}$ and $r$ are the central positions and radii of the ink stones. Eq. (4) describes that the action Noutan consists of (1) a cross-like translation, dipping and scraping in $4$ directions for fully moistening and adjusting the brush in the light ink, and (2) a translation and dipping for mixing in the thick ink. Next, the dipping time $T_{dip}$ and dipping height $L_{dip}$ of the brush in the thick ink can be applied to control the shading degree of Noutan:

where $L_{brush}$ is the length of a brush, and $c_{1}$ is a constant coefficient. Let $\lambda=L_{dip}/L_{brush}$ which denotes the dipped ratio of a brush. Then, a small $\lambda$ and a small $T_{dip}$ will create bright shading effect as shown in Fig. 10(a) because of shallow mixing in the thick ink, and a large $\lambda$ and a large $T_{dip}$ will generate dark shading effect as shown in Fig. 10(b) because of sufficient mixing in the thick ink.

Kasure mainly depends on the property of brush hairs, the quantity of ink absorbed by the brush and the area of drawing strokes. It is difficult to dynamically control the property of each brush hair such as the length, stiffness and the ability of absorbing fluid. However, it is possible to control the amount of ink on the brush and the area of strokes. Therefore, we assume that the degree of scratchiness is proportional to the stroke area which consists of a series of trapezoids and inversely proportional to the absorbed quantity of ink which is related to the dipping time $T_{dip}$ and the dipped ratio $\lambda$:

where $c_{2}$ and $c_{3}$ are constant coefficients. In our experiment, the brush remained dry before being dipped into an ink stone. Fig. 10(c) shows the distinguishable effect of scratchiness when setting a small $\lambda$ and a small $T_{dip}$. In contrast, the Kasure effect only happened at the very end of the stroke when setting a large $\lambda$ and a large $T_{dip}$ as shown in Fig. 7(f).

In order to draw stylized strokes using robotic arms, we first mapped the sampling positions and thicknesses $(C_{i},t_{i})(i=1,2,...,n)$ of an optimized stroke in simulation space to $(C^{\prime}_{i},h_{i})$ in robot Cartesian space as introduced in Sec. VI. Then, we set the painting styles, dipping time $T_{dip}$ and dipped ratio $\lambda$ of a brush for stylized drawing. Next, the brush was lifted to an initial safe plane with a height $h_{0}$ above a drawing plane waiting for real executions as shown in Fig. 9. After that, a series of actions such as $A_{tran}$, $A_{dip}$ and $A_{scrape}$ were combined to form various painting styles. Finally, the stroke trajectory $(C^{\prime}_{i},h_{i})$ integrated with the way-points of these actions were automatically converted to joints’ rotations from Cartesian space to joint space by solving an inverse kinematics problem [35]. One of an important design of trajectory planning is to choose a joint-space or task-space trajectory. In our framework, we used task-space trajectory since it generates physical strokes more faithful to the simulated ones. Furthermore, the speed of robotic drawing for each stroke was constrained by a typical trapezoidal velocity model which ensures piecewise trajectories of constant acceleration, zero acceleration, and constant deceleration.

Fig. 11 shows two different simplified contours generated from the input rough contours of a 3D model. We can observe that our updated model with augmented dataset generated cleaner and longer contours than the results of the pre-trained model of Simo-Serra et al. [31].

If we directly use all the simplified contours to generate strokes without interactive editing, it will generate many redundant strokes overlapping with each other or some expressive strokes might be ignored as shown in Fig. 12(c). Therefore, we designed the interactive editing tools to choose and modify expressive strokes by referring to both the simplified contours and the original 3D models. Fig. 12(d) shows that our editing tools can generate clean and expressive strokes.

To evaluate the effectiveness of stroke optimization algorithm, we have conducted an experiment to verify that the physically drawing stroke can keep the shape of a simulated curve when the brush is deformed with various descent height $h_{i}$ as shown in Fig. 13. The first row of Fig. 13 show the robotic drawing results of strokes before B-spline fitting and optimization. The second row of Fig. 13 shows the robotic drawing results of strokes after B-spline fitting without optimization, and the third row of Fig. 13 shows the robotic drawing results of strokes after B-spline fitting with optimization.

To quantitatively evaluate the stroke optimization method, we utilized the angle of contingence $\theta$ (or the slope change between contiguous straight-line segments at a point) presented by [36] to represent the discrete curvature at the $6$ typical turning points of the simulated curve as shown in Fig. 14. The original angles of contingence of the turning points in the simulated curve are $135^{\circ}$, $93^{\circ}$, $130^{\circ}$, $81^{\circ}$, $148^{\circ}$ and $150^{\circ}$. The angles of contingence corresponding to the three kinds of physically drawing strokes in Fig. 13 are shown in Tab. II, III and IV. Fig. 15(a-c) show the corresponding changing trends of $\theta$ from Tab. II, III and IV.

From Tab. II and III, we can observe that the total average error of $\theta$ after fitting is $16.784$, that is much larger than that of angle of contingence ($10.632$) before fitting, though the fitted curve generated more smooth strokes as shown in Fig. 13. From Tab. II and IV, we can conclude that the optimized B-spline curve not only create the minimal total average error $8.83$, but also generate smooth strokes. Therefore, the proposed B-spline fitting and optimization method has the merits of retaining the shape of the simulated stroke as well as generating physically smooth strokes.

Furthermore, we have compared the drawing results of the “Stanford Bunny” before and after using the optimization method. It can be observed that the optimized strokes are more smooth and natural than the original strokes before optimization as shown in the colored rectangles of Fig. 16.

To test the oriental ink painting styles, we have conducted an experiment to show the effect of Noutan and Kasure on “Stanford Bunny” and “The Utah Teapot” from different viewpoints. Fig. 17(a) shows the gradient shading effect of Noutan style and Fig. 17(b) shows the scratchy effect of Kasure style. The extreme case of Noutan style is using thick ink or light ink only to convey strong or delicate impression as shown in Fig. 17(c, d).

To quantitatively measure the intensity of the two ink painting styles, we set $12$ different combinations of dipped ratio $\lambda$ and dipping time $T_{dip}$, and then draw the painting with Noutan and Kasure styles on rice paper, and $2$ experts were invited to score each painting.

Fig. 18 shows the results of Noutan experiment. To observe the different effect of parameters, we changed one parameter while keeping the remaining parameter constant. The first row of Fig. 18 shows the score of Noutan effect when $\lambda$ is constant and $T_{dip}$ is changing, and the second row of Fig. 18 shows the score of Noutan effect when $T_{dip}$ is constant and $\lambda$ is changing. As the dipping time is gradually increased from $0.5s$ to $3.0s$, the Noutan intensity decreases and converges to $0.6$. On the other hand, when the dipped ratio is increased from $2/12$ to $7/12$, the Noutan intensity gradually decreases from $0.8$ to $0.2$.

Fig. 19 shows the results of Kasure experiment. The relationship of the dipping time and the dipped ratio are similar to the $Noutan$ experiment. Moreover, we can observe that the dipped ratio has a greater impact on style intensity than the dipping time.

Furthermore, we have designed an experiment to demonstrate the effect of robot’s velocity on Kasure style of a stroke and the drawing results are shown in Figure 20. The experiment was conducted with the dipped ratio $\lambda=1/3$ and the dipping time $T_{dip}=2.0s$. The drawing velocity $v$ was set from $50mm/s$ to $110mm/s$ with an increasing step $15mm/s$. From the results, we can observe that the increasing of robot’s velocity will result in more intensive Kasure effect at the initial stage from $50mm/s$ to $95mm/s$. However, the Kasure effect will converge as the speed increases at the closing stage from $95mm/s$ to $110mm/s$. The results showed that robot’s velocity does impact the final painting styles.

However, there are many factors possibly impact the painting styles such as the drawing speed, the property of brush hairs, the quantity of ink absorbed by the brush and the area of drawing strokes. In our work, it is difficult to consider all these factors for controlling a specific painting style. Inspired by Strassmann’s pioneer work of non-photorealistic rendering “Hairy Brushes”, we fixed the drawing velocity and mainly consider the factor ink quantity because the ink supply on each bristle of a brush is assumed to be a reservoir of a finite quantity of fluid and the quantity is decreased as the brush moves through a stroke [34]. Therefore, our framework chooses to generate ink style by controlling ink quantity instead of robot’s velocity.

We also examined the ability of our framework to draw complicated stylized portrait from the standard 3D model “Lucy” as shown in Fig. 21. The results showed that the physically drawing strokes retained the characteristics of reference model after feature extraction, simplification, digital simulation and optimization. Note that the Kasure and Noutan styles have been applied to the hair and shoulder of “Lucy” respectively.

To evaluate the scalability of our robotic drawing framework for arbitrary user-defined model, we collected approximately 40 photos around a sculpture named “Youren” by a normal smartphone, and then reconstructed its point cloud and surface by using SfM (Structure from Motion) and screened Poisson reconstruction algorithms. Fig. 22 shows that a small number of stylized strokes can depict the facial expression of a character, which indicates that our framework has the potential for drawing vivid portraits from varying viewpoints once a 3D model is given.

We calculated the time of line extraction, stroke optimization and robotic drawing of the tested $4$ 3D models, each of which has 2 viewpoints, and the quantitative results of our robotic drawing framework is shown in Tab. V.

Although strokes have various length and thickness, the average execution time of line extraction, stroke optimization and robotic drawing of each stroke is $8.61s$, $0.31s$ and $15.04s$. The results showed that the robotic drawing of stroke is the most time-consuming stage because the physical drawing of a stroke with a brush is inherently slow, and the expressive line extraction is the second time-consuming stage because it requires user interaction, and the stroke optimization is the most efficient stage because it is fully automatic.