FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environment

Stroke-Based Rendering (SBR) recreates a given target image using a set of primitive elements that usually resemble brush strokes of paint. Procedural SBR methods generally use rules and heuristics to generate the stroke plan [13, 14]. Planning-based SBR methods use search, optimization, or learning models such as Reinforcement Learning or Recurrent Neural Networks to generate a stroke plan with an objective of replicating an input image [15, 16].

Recent SBR methods expands the input space to incorporate high-level goals to generate brush stroke simulated paintings based on language descriptions and/or style specification [10, 9, 17]. While these methods present appreciable results in simulation, technical challenges specific to transitioning from simulation to real robots have not been addressed.

There have been numerous robot-created paintings including notable works that had competed in an annual competition in 2016–2018 [18], but technical details of most works have not been published. Based on published works, existing robot painting approaches can roughly be categorized into two groups: engineered systems and learning-enabled systems.

Brush strokes can be represented using a height map and a color map as in [24] where Generative Adversarial Networks are used to map a user input trajectory into a synthesized brush stroke. In their work, both training and testing were done using data synthesized using a volumetric oil painting simulator based on WetBrush [25]. While the outputs appear impressive in simulation, the challenge still remains unanswered how such a simulated input can be translated into a real painting, for example, by a robot.

Wang et al. [26] use brush parameters such as the width, drag, and offset of the brush’s bristles to create a very accurate brush stroke model. They use pseudospectral optimal control to optimize trajectories of brush strokes to fit the target calligraphy character, which works well with calligraphy where an initial path is given in a reasonably accurate form and the brush strokes are clearly separated by white space. In the painting domain, however, a more generalizable approach is needed due to the fact that the shapes of brush strokes used in painting are highly flexible and unconstrained and that brush strokes frequently overlap with previous ones.

Inspired by [26], we parameterize the space of brush strokes using three parameters as shown in Fig. 5. In addition to brush shape attributes, i.e., the length $l$ of the stroke, and the amount $b$ that the stroke bends up or down, the thickness $h$ of the stroke specifies how far the brush is pressed proportionally to the canvas. A brush stroke is parameterized by its shape, denoted by ($h,l,b$), the location coordinates on a canvas ($x,y$), orientation $\theta$, and color $\rho$ in the RGB format. The stroke trajectory can then be represented by a cubic Bézier curve where the horizontal coordinates are a linear interpolation between $0$ and $l$, and the vertical coordinates are $0$ at the end points and $b$ in the center points.

Whereas existing models such as [26] use only shape features of the rendered images that would require some model of a brush tool for a robot control interface, our definition of thickness connects the parameter space with a brush tool and a robot. During the calibration phase, we generate random brush strokes to train the param2stroke model, a Neural Network comprised of two linear layers followed by two convolutional layers and an bilinear upscaler, that translates a brush stroke shape tuple directly into the appearance map of the brush stroke. The brush stroke shape tuple can deterministically be translated into control inputs for a real robot.

A rudimentary approach to creating a differentiable, simulated robot painting environment would be to allow the robot to paint randomly and continuously to collect a large enough dataset of paired robot actions to the effects on the canvas to model this relationship. While this method works well in simulated environments [24, 21, 16, 17], where thousands of brush strokes can be produced on the order of seconds, generating a similarly large-sized real dataset is impractical. Painting in real life is slow, and if the brush or other materials were altered, the entire process would need to be restarted. Instead, we augment the dataset using existing differentiable functions, such as rigid transformations for positioning and orienting strokes and stamping methodology for rendering individual brush strokes onto an existing canvas, to allow our painting environment to be simulated with a reasonably small number of real brush strokes for modeling.

The stroke rendering process in our simulation is depicted in Fig. 4: the param2stroke network translates the thickness, bend, and length parameters into a 2d magnitude map of the brush stroke’s predicted appearance. This magnitude map is then padded such that it is the size of a full canvas. Then the map is translated and rotated to the specified orientation and location. The magnitude map is converted into an RGBA image, and then the stroke is applied to a given existing canvas. Strokes can be layered upon each other to create a complete simulated painting. They can also be rendered onto a photograph of the real canvas for planning throughout the painting process. The whole rendering process is differentiable, meaning that the loss value computed using the rendered canvas can be differentiated, back-propagated through the simulator, and a Stochastic Gradient Descent algorithm updates the brush stroke parameters such that the parameters minimize the loss function.

Algorithm 1 sketches our planning algorithm for painting with visual feedback given high-level semantic goals. The optimize function compares the goals (image or text) with the simulated painting according to one or more objective functions detailed in Section III-E. The simulated painting is the rendering of the brush stroke plan onto the current canvas–i.e., the current image observed by the camera sensor.

To start the paint function, $n\_strokes$ brush strokes are randomly initialized by sampling uniformly over the brush stroke parameters. An initial pass at optimization can be performed with initial high-level goal(s), then the plan is optimized to use the full objective(s). The first $batch\_size$ brush strokes from the plan are executed, then the remainder of the plan is re-optimized based on the executed brush strokes and objective(s).

To enable our system to achieve high-level goals, we employ a variety of objective functions from recent image synthesis literature. Each objective function has a loss function that compares the brush stroke plan ($p$) to the target input ($t$) which may be language or an image. A plan $p_{next}$ for the next time step is rendered into a raster image using a differentiable simulated environment ($r$). These objective functions can be used in different combinations to achieve high-level, artistic tasks, e.g., painting from language description with or without a specified style, painting images conceptually, or painting from a sketch.

The trained param2stroke model can produce strokes with continuous parameter values as seen in Fig.6. Fig. 7 shows the difference between real brush strokes and FRIDA’s modeled brush strokes using the same input parameters. We also compare these strokes to DiffVG [32] which was used for brush stroke planning in [22]. The average $L_{1}$ loss between the modeled and real strokes was significantly (p-value $<.01$) less for FRIDA’s stroke model than DiffVG.

We qualitatively compared our approach to Huang et al. 2019 [16] and Schaldenbrand & Oh 2021 [21]. Fig. 8 compares the brush strokes in early stages of painting simulation where we can observe drastic differences.

Fig. 9 shows the comparison in terms of the sim2real gap for entire paintings. In simulation, Huang et al. 2019’s Reinforcement Learning (RL) model is able to almost perfectly replicate a given image due to their unconstrained stroke model, e.g., allowing strokes that are huge in size and have varying opacity; however, when we fed the strokes to a painting robot, the painting produced was vastly dissimilar to both the simulation and target image. Schaldenbrand & Oh 2021 constrained the brush stroke parameters (length, width, color, and opacity) such that a robot was more capable of executing the strokes; however, the constraints made it challenging for their RL model to accurately replicate a target image. For the proposed approach, we used our simulation to recreate the target image using the Simple Replication Objective (Eq. 3) and did not re-plan with perception for fair comparison. Our proposed approach shows clearly visible improvement in recreation both in simulation and real painting.

We painted with and without FRIDA’s dynamic replanning system and plotted the deviation from the initial plan in Fig. 11. Without replanning, the difference between the current and initial plan grows linearly as the plan is executed from simulation to reality stroke by stroke. With replanning, the plan changes more significantly from the initial plan as the algorithm adapts to the stochastic execution of the plan, resembling the creative process of human artists [1].