publications

Robust state estimation for highly dynamic motion of legged robots remains challenging, especially in dynamic, contact-rich scenarios. Traditional approaches often rely on binary contact states that fail to capture the nuances of partial contact or directional slippage. This paper presents CoCo-InEKF, a differentiable invariant extended Kalman filter that utilizes continuous contact velocity covariances instead of binary contact states. These learned covariances allow the method to dynamically modulate contact confidence, accounting for more nuanced conditions ranging from firm contact to directional slippage or no contact. To predict these covariances for a set of predefined contact candidate points, we employ a lightweight neural network trained end-to-end using a state-error loss. This approach eliminates the need for heuristic ground-truth contact labels. In addition, we propose an automated contact candidate selection procedure and demonstrate that our method is insensitive to their exact placement. Experiments on a bipedal robot demonstrate a superior accuracy-efficiency tradeoff for linear velocity estimation, as well as improved filter consistency compared to baseline methods. This enables the robust execution of challenging motions, including dancing and complex ground interactions – both in simulation and in the real world.

Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.

We present Kamino, a GPU-based physics solver for massively parallel simulations of heterogeneous highly-coupled mechanical systems. Implemented in Python using NVIDIA Warp and integrated into the Newton framework, it enables the application of data-driven methods, such as large-scale reinforcement learning, to complex robotic systems that exhibit strongly coupled kinematic and dynamic constraints such as kinematic loops. The latter are often circumvented by practitioners; approximating the system topology as a kinematic tree and incorporating explicit loop-closure constraints or so-called mimic joints. Kamino aims at alleviating this burden by natively supporting these types of coupling. This capability facilitates high-throughput parallelized simulations that capture the true nature of mechanical systems that exploit closed kinematic chains for mechanical advantage. Moreover, Kamino supports heterogeneous worlds, allowing for batched simulation of structurally diverse robots on a single GPU. At its core lies a state-of-the-art constrained optimization algorithm that computes constraint forces by solving the constrained rigid multi-body forward dynamics transcribed as a nonlinear complementarity problem. This leads to high-fidelity simulations that can resolve contact dynamics without resorting to approximate models that simplify and/or convexify the problem. We demonstrate RL policy training on DR Legs, a biped with six nested kinematic loops, generating a feasible walking policy while simulating 4096 parallel environments on a single GPU.

Animated characters often move in non-physical ways and have proportions that are far from a typical walking robot. This provides an ideal platform for innovation in both mechanical design and stylized motion control. In this paper, we bring Olaf to life in the physical world, relying on reinforcement learning guided by animation references for control. To create the illusion of Olaf's feet moving along his body, we hide two asymmetric legs under a soft foam skirt. To fit actuators inside the character, we use spherical and planar linkages in the arms, mouth, and eyes. Because the walk cycle results in harsh contact sounds, we introduce additional rewards that noticeably reduce impact noise. The large head, driven by small actuators in the character's slim neck, creates a risk of overheating, amplified by the costume. To keep actuators from overheating, we feed temperature values as additional inputs to policies, introducing new rewards to keep them within bounds. We validate the efficacy of our modeling in simulation and on hardware, demonstrating an unmatched level of believability for a costumed robotic character.

Despite recent advances in robust locomotion, bipedal robots operating in the real world remain at risk of falling. While most research focuses on preventing such events, we instead concentrate on the phenomenon of falling itself. Specifically, we aim to reduce physical damage to the robot while providing users with control over a robot's end pose. To this end, we propose a robot agnostic reward function that balances the achievement of a desired end pose with impact minimization and the protection of critical robot parts during reinforcement learning. To make the policy robust to a broad range of initial falling conditions and to enable the specification of an arbitrary and unseen end pose at inference time, we introduce a simulation-based sampling strategy of initial and end poses. Through simulated and real-world experiments, our work demonstrates that even bipedal robots can perform controlled, soft falls.

Teleoperated robotic characters can perform expressive interactions with humans, relying on the operators’ experience and social intuition. In this work, we propose to create autonomous interactive robots, by training a model to imitate operator data. Our model is trained on a dataset of human-robot interactions, where an expert operator is asked to vary the interactions and mood of the robot, while the operator commands as well as the pose of the human and robot are recorded. Our approach learns to predict continuous operator commands through a diffusion process and discrete commands through a classifier, all unified within a single transformer architecture. We evaluate the resulting model in simulation and with a user study on the real system. We show that our method enables simple autonomous human-robot interactions that are comparable to the expert-operator baseline, and that users can recognize the different robot moods as generated by our model. Finally, we demonstrate a zero-shot transfer of our model onto a different robotic platform with the same operator interface.

Dynamic locomotion of legged robots is a critical yet challenging topic in expanding the operational range of mobile robots. It requires precise planning when possible footholds are sparse, robustness against uncertainties and disturbances, and generalizability across diverse terrains. Although traditional model-based controllers excel at planning on complex terrains, they struggle with real-world uncertainties. Learning-based controllers offer robustness to such uncertainties but often lack precision on terrains with sparse steppable areas. Hybrid methods achieve enhanced robustness on sparse terrains by combining both methods but are computationally demanding and constrained by the inherent limitations of model-based planners. To achieve generalized legged locomotion on diverse terrains while preserving the robustness of learning-based controllers, this paper proposes an attention-based map encoding conditioned on robot proprioception, which is trained as part of the controller using reinforcement learning. We show that the network learns to focus on steppable areas for future footholds when the robot dynamically navigates diverse and challenging terrains. We synthesized behaviors that exhibited robustness against uncertainties while enabling precise and agile traversal of sparse terrains. In addition, our method offers a way to interpret the topographical perception of a neural network. We have trained two controllers for a 12-degrees-of-freedom quadrupedal robot and a 23-degrees-of-freedom humanoid robot and tested the resulting controllers in the real world under various challenging indoor and outdoor scenarios, including ones unseen during training. We achieved generalized legged locomotion controllers via reinforcement learning and attention-based map encoding.

Reinforcement learning (RL) has significantly advanced the control of physics-based and robotic characters that track kinematic reference motion. However, methods typically rely on a weighted sum of conflicting reward functions, requiring extensive tuning to achieve a desired behavior. Due to the computational cost of RL, this iterative process is a tedious, time-intensive task. Furthermore, for robotics applications, the weights need to be chosen such that the policy performs well in the real world, despite inevitable sim-to-real gaps. To address these challenges, we propose a multi-objective reinforcement learning framework that trains a single policy conditioned on a set of weights, spanning the Pareto front of reward trade-offs. Within this framework, weights can be selected and tuned after training, significantly speeding up iteration time. We demonstrate how this improved workflow can be used to perform highly dynamic motions with a robot character. Moreover, we explore how weight-conditioned policies can be leveraged in hierarchical settings, using a high-level policy to dynamically select weights according to the current task. We show that the multi-objective policy encodes a diverse spectrum of behaviors, facilitating efficient adaptation to novel tasks.

We present a constrained Rigid Body Dynamics (RBD) that guarantees satisfaction of kinematic constraints, enabling direct simulation of complex mechanical systems with arbitrary kinematic structures. To ensure constraint satisfaction, we use an implicit integration scheme. For this purpose, we derive compatible dynamic equations expressed through the quaternion time derivative, adopting an additive approach to quaternion updates instead of a multiplicative one, while enforcing quaternion unit-length as a constraint. We support all joints between rigid bodies that restrict subsets of the three translational or three rotational degrees of freedom, including position- and force-based actuation. Their constraints are formulated such that Lagrange multipliers are interpretable as joint forces and torques. We discuss a unified solution strategy for systems with redundant constraints, overactuation, and passive degrees of freedom, by eliminating redundant constraints and navigating the subspaces spanned by multipliers. As our method uses a standard additive update, we can interface with unconditionally-stable implicit integrators. Moreover, the simulation can readily be made differentiable as we show with examples.

This technical report provides an in-depth evaluation of both established and state-of-the-art methods for simulating constrained rigid multi-body systems with hard-contact dynamics, using formulations of Nonlinear Complementarity Problems (NCPs). We are particularly interest in examining the simulation of highly coupled mechanical systems with multitudes of closed-loop bilateral kinematic joint constraints in the presence of additional unilateral constraints such as joint limits and frictional contacts with restitutive impacts. This work thus presents an up-to-date literature survey of the relevant fields, as well as an in-depth description of the approaches used for the formulation and solving of the numerical time-integration problem in a maximal coordinate setting. More specifically, our focus lies on a version of the overall problem that decomposes it into the forward dynamics problem followed by a time-integration using the states of the bodies and the constraint reactions rendered by the former. We then proceed to elaborate on the formulations used to model frictional contact dynamics and define a set of solvers that are representative of those currently employed in the majority of the established physics engines. A key aspect of this work is the definition of a benchmarking framework that we propose as a means to both qualitatively and quantitatively evaluate the performance envelopes of the set of solvers on a diverse set of challenging simulation scenarios. We thus present an extensive set of experiments that aim at highlighting the absolute and relative performance of all solvers on particular problems of interest as well as aggravatingly over the complete set defined in the suite.

Recent advancements in generative motion models have achieved remarkable results, enabling the synthesis of lifelike human motions from textual descriptions. These kinematic approaches, while visually appealing, often produce motions that fail to adhere to physical constraints, resulting in artifacts that impede real-world deployment. To address this issue, we introduce a novel method that integrates kinematic generative models with physics-based character control. Our approach begins by training a reward surrogate to predict the performance of the downstream non-differentiable control task, offering an efficient and differentiable loss function. This reward model is then employed to fine-tune a baseline generative model, ensuring that the generated motions are not only diverse but also physically plausible for real-world scenarios. The outcome of our processing is the Robot Motion Diffusion Model (RobotMDM), a text-conditioned kinematic diffusion model that interfaces with a reinforcement learning-based tracking controller. We demonstrate the effectiveness of this method on a challenging humanoid robot, confirming its practical utility and robustness in dynamic environments.

We introduce Spline-based Transformers, a novel class of Transformer models that eliminate the need for positional encoding. Inspired by workflows using splines in computer animation, our Spline-based Transformers embed an input sequence of elements as a smooth trajectory in latent space. Overcoming drawbacks of positional encoding such as sequence length extrapolation, Spline-based Transformers also provide a novel way for users to interact with transformer latent spaces by directly manipulating the latent control points to create new latent trajectories and sequences. We demonstrate the superior performance of our approach in comparison to conventional positional encoding on a variety of datasets, ranging from synthetic 2D to large-scale real-world datasets of images, 3D shapes, and animations.

Recent progress in physics-based character control has made it possible to learn policies from unstructured motion data. However, it remains challenging to train a single control policy that works with diverse and unseen motions, and can be deployed to real-world physical robots. In this paper, we propose a two-stage technique that enables the control of a character with a full-body kinematic motion reference, with a focus on imitation accuracy. In a first stage, we extract a latent space encoding by training a variational autoencoder, taking short windows of motion from unstructured data as input. We then use the embedding from the time-varying latent code to train a conditional policy in a second stage, providing a mapping from kinematic input to dynamics-aware output. By keeping the two stages separate, we benefit from self-supervised methods to get better latent codes and explicit imitation rewards to avoid mode collapse. We demonstrate the efficiency and robustness of our method in simulation, with unseen user-specified motions, and on a bipedal robot, where we bring dynamic motions to the real world.

Procedural animation has seen widespread use in the design of expressive walking gaits for virtual characters. While similar tools could breathe life into robotic characters, existing techniques are largely unaware of the kinematic and dynamic constraints imposed by physical robots. In this paper, we propose a system for the artist-directed authoring of stylized bipedal walking gaits, tailored for execution on robotic characters. The artist interfaces with an interactive editing tool that generates the desired character motion in realtime, either on the physical or simulated robot, using a model-based control stack. Each walking style is encoded as a set of sample parameters which are translated into whole-body reference trajectories using the proposed procedural animation technique. In order to generalize the stylized gait over a continuous range of input velocities, we employ a phase-space blending strategy that interpolates a set of example walk cycles authored by the animator while preserving contact constraints. To demonstrate the utility of our approach, we animate gaits for a custom, free-walking robotic character, and show, with two additional in-simulation examples, how our procedural animation technique generalizes to bipeds with different degrees of freedom, proportions, and mass distributions.

We propose a computational design pipeline for pneumatically-actuated soft robots interacting with their environment through contact. We optimize the shape of the robot with a shape optimization approach, using a physically-accurate high-order finite element model for the forward simulation. Our approach enables fine-grained control over both deformation and contact forces by optimizing the shape of internal cavities, which we exploit to design pneumatically-actuated robots that can assume user-prescribed poses, or apply user-controlled forces. We demonstrate the efficacy of our method on two artistic and two functional examples.

Legged robots have achieved impressive feats in dynamic locomotion in challenging unstructured terrain. However, in entertainment applications, the design and control of these robots face additional challenges in appealing to human audiences. This work aims to unify expressive, artist-directed motions and robust dynamic mobility for legged robots. To this end, we introduce a new bipedal robot, designed with a focus on character-driven mechanical features. We present a reinforcement learning-based control architecture to robustly execute artistic motions conditioned on command signals. During runtime, these command signals are generated by an animation engine which composes and blends between multiple animation sources. Finally, an intuitive operator interface enables real-time show performances with the robot.

The kinematic motion of a robotic character is defined by its mechanical joints and actuators that restrict the relative motion of its rigid components. Designing robots that perform a given target motion as closely as possible with a fixed number of actuated degrees of freedom is challenging, especially for robots that form kinematic loops. In this paper, we propose a technique that simultaneously solves for optimal design and control parameters for a robotic character whose design is parameterized with configurable joints. At the technical core of our technique is an efficient solution strategy that uses dynamic programming to solve for optimal state, control, and design parameters, together with a strategy to remove redundant constraints that commonly exist in general robot assemblies with kinematic loops. We demonstrate the efficacy of our approach by either editing the design of an existing robotic character, or by optimizing the design of a new character to perform a desired motion.

Legged robots are designed to perform highly dynamic motions. However, it remains challenging for users to retarget expressive motions onto these complex systems. In this paper, we present a Differentiable Optimal Control (DOC) framework that facilitates the transfer of rich motions from either animals or animations onto these robots. Interfacing with either motion capture or animation data, we formulate retargeting objectives whose parameters make them agnostic to differences in proportions and numbers of degrees of freedom between input and robot. Optimizing these parameters over the manifold spanned by optimal state and control trajectories, we minimize the retargeting error. We demonstrate the utility and efficacy of our modeling by applying DOC to a Model-Predictive Control (MPC) formulation, showing retargeting results for a family of robots of varying proportions and mass distribution. With a hardware deployment, we further show that the retargeted motions are physically feasible, while MPC ensures that the robots retain their capability to react to unexpected disturbances.

Simulation representations of robots have advanced in recent years. Yet, there remain significant sim-to-real gaps because of modeling assumptions and hard-to-model behaviors such as friction. In this letter, we propose to augment common simulation representations with a transformer-inspired architecture, by training a network to predict the true state of robot building blocks given their simulation state. Because we augment building blocks, rather than the full simulation state, we make our approach modular which improves generalizability and robustness. We use our neural network to augment the state of robot actuators, and also of rigid body states. Our actuator augmentation generalizes well across robots, and our rigid body augmentation results in improvements even under high uncertainty in model parameters.

Robot mechanisms that exploit compliance can perform complex tasks under uncertainty using simple control strategies, but it remains difficult to design mechanisms with a desired embodied intelligence. In this article, we propose an automated design technique that optimizes the desired load-displacement behavior of planar flexible-link mechanisms. To do so, we replace a subset of rigid with flexible links in an existing mechanism, and optimize their rest shape. We demonstrate the efficacy of our approach on a set of examples, including two fabricated prototypes, illustrating applications for grasping and locomotion tasks.

Embodied intelligence (intelligence that requires and leverages a physical body) is a well-known paradigm in soft robotics, but its mathematical description and consequent computational modelling remain elusive, with a need for models that can be used for design and control purposes. We argue that filling this gap will enable full uptake of embodied intelligence in soft robots. We provide a concise guide to the main mathematical modelling approaches, and consequent computational modelling strategies, that can be used to describe soft robots and their physical interactions with the surrounding environment, including fluid and solid media. We aim to convey the challenges and opportunities within the context of modelling the physical interactions underpinning embodied intelligence. We emphasize that interdisciplinary work is required, especially in the context of fully coupled robot–environment interaction modelling. Promoting this dialogue across disciplines is a necessary step to further advance the field of soft robotics.

We introduce a singularity-aware design optimization method for spatial multi-degree-of-freedom mechanical linkages. At the core of our approach is an adversarial sampling strategy, which actively detects singular configurations within the targeted operation range. The detection of singularities in both forward and inverse kinematics allows for two-way bijective mappings between input and output trajectories on our optimized designs, thus enabling robust control. We demonstrate our approach on a set of simulation examples and provide additional validation on physical prototypes.

This paper presents an interactive design system that allows the user to create and fabricate stylized sculptures in water-based clay, using a standard 6-axis robot arm. This system facilitates the materialization of abstract design intentions into clay, through the algorithmic formulation of sculpting styles, the optimal path planning of the sculpting toolpaths, and a subtractive robotic fabrication process using customized tools. Unlike other precision-driven fabrication technologies, the authors embrace artistic uncertainty by conducting manual and robotic sculpting experiments and incorporating prominent parameters that affect the fabrication quality. The versatility of the described approach is demonstrated by designing a series of sculpting styles over a wide range of 3D models and robotically fabricating them in clay. Additionally, the paper explores various strategies for designing stylized robotic sculpting patterns by generating toolpaths informed by different techniques.

Purpose of Review We discuss the use of differentiable simulation for computational problems in soft robotics. This includes characterizing the mechanical behavior of soft robots, optimally controlling embedded soft actuators or active materials, and estimating the robot’s state from readings of embedded sensors. Moreover, we discuss how design optimization can help to optimally place soft actuators and sensors. Recent Findings We expatiate on the adoption of simulation and optimization tools in the process of designing and controlling soft robots. We include a discussion of rigid-flexible systems and the use of differentiable simulation in combination with machine learning. Summary We review the state of the art in the computational modeling of soft robots and provide a summary of the required mathematical tools. We also review several open questions where computation could help to move the field forward, and discuss the role of differentiable simulation in managing the ever-growing design complexity of next-generation soft robots.

Robots with kinematic loops are known to have superior mechanical performance. However, due to these loops, their modeling and control is challenging, and prevents a more widespread use. In this letter, we describe a versatile Inverse Kinematics (IK) formulation for the retargeting of expressive motions onto mechanical systems with loops. We support the precise control of the position and orientation of several end-effectors, and the Center of Mass (CoM) of slowly walking robots. Our formulation safeguards against a disassembly when IK targets are moved outside the workspace of the robot, and we introduce a regularizer that smoothly circumvents kinematic singularities where velocities go to infinity. With several validation examples and three physical robots, we demonstrate the versatility and efficacy of our IK on overactuated systems with loops, and for the retargeting of an expressive motion onto a bipedal robot.

This article introduces a computational approach for routing thin artificial muscle actuators through hyperelastic soft robots, in order to achieve a desired deformation behavior. Provided with a robot design and a set of example deformations, we continuously co-optimize the routing of actuators, and their actuation, to approximate example deformations as closely as possible. We introduce a data-driven model for McKibben muscles, modeling their contraction behavior when embedded in a silicone elastomer matrix. To enable the automated routing, a differentiable hyperelastic material simulation is presented. Because standard finite elements are not differentiable at element boundaries, we implement a moving least squares formulation, making the deformation gradient twice differentiable. Our robots are fabricated in a two-step molding process, with the complex mold design steps automated. While most soft robotic designs utilize bending, we study the use of our technique in approximating twisting deformations on a bar example. To demonstrate the efficacy of our technique in soft robotic design, we show a continuum robot, a tentacle, and a four-legged walking robot.

We present an interactive design system that allows users to create sculpting styles and fabricate clay models using a standard 6-axis robot arm. Given a general mesh as input, the user iteratively selects sub-areas of the mesh through decomposition and embeds the design expression into an initial set of toolpaths by modifying key parameters that affect the visual appearance of the sculpted surface finish. These parameters were identified and extracted through a series of design experiments, using a customized loop tool to cut the water-based clay material. The initialized toolpaths are fed into the optimization component of our system afterwards for optimal path planning, aiming to find the robotic sculpting motions that match the target surface, maintaining the design expression, and resolving collisions and reachability issues. We demonstrate the versatility of our approach by designing and fabricating different sculpting styles over a wide range of clay models.

We present a differentiable dynamics solver that is able to handle frictional contact for rigid and deformable objects within a unified framework. Through a principled mollification of normal and tangential contact forces, our method circumvents the main difficulties inherent to the non-smooth nature of frictional contact. We combine this new contact model with fully-implicit time integration to obtain a robust and efficient dynamics solver that is analytically differentiable. In conjunction with adjoint sensitivity analysis, our formulation enables gradient-based optimization with adaptive trade-offs between simulation accuracy and smoothness of objective function landscapes. We thoroughly analyse our approach on a set of simulation examples involving rigid bodies, visco-elastic materials, and coupled multi-body systems. We furthermore showcase applications of our differentiable simulator to parameter estimation for deformable objects, motion planning for robotic manipulation, trajectory optimization for compliant walking robots, as well as efficient self-supervised learning of control policies.

We present a practical neural computational approach for interactive design of animatronic facial performances. An offline quasi-static reference simulation accurately predicts hyperelastic skin deformations, driven by a coupled mechanical assembly. To achieve interactive digital pose design, we train a shallow fully connected neural network (KSNN) on input motor activations to solve the simulated mesh vertex positions. Our fully automatic synthetic training algorithm enables a first-of-its-kind Active Learning framework (GEN-LAL) for generative modeling of facial pose simulations. With adaptive selection we significantly reduce training time to within half that of the unmodified training approach for each new animatronic figure.

We present a computational technique that aids with the design of structurally‐sound metal frames, tailored for robotic fabrication using an existing process that integrates automated bar bending, welding, and cutting. Aligning frames with structurally‐favorable orientations, and decomposing models into fabricable units, we make the fabrication process scale‐invariant, and frames globally align in an aesthetically‐pleasing and structurally‐informed manner. Relying on standard analysis of frames, we then co‐optimize the shape and topology of bars at the local unit level. At this level, we minimize combinations of functional and aesthetic objectives under strict fabrication constraints that model the assembly of discrete sets of bent bars. We demonstrate the capabilities of our global‐to‐local approach on four robotically‐constructed examples.

Soft robots have applications in safe human–robot interactions, manipulation of fragile objects, and locomotion in challenging and unstructured environments. In this article, we present a computational method for augmenting soft robots with proprioceptive sensing capabilities. Our method automatically computes a minimal stretch-receptive sensor network to user-provided soft robotic designs, which is optimized to perform well under a set of user-specified deformation-force pairs. The sensorized robots are able to reconstruct their full deformation state, under interaction forces. We cast our sensor design as a subselection problem, selecting a minimal set of sensors from a large set of fabricable ones, which minimizes the error when sensing specified deformation-force pairs. Unique to our approach is the use of an analytical gradient of our reconstruction performance measure with respect to selection variables. We demonstrate our technique on a bending bar and gripper example, illustrating more complex designs with a simulated tentacle.

Simulation-driven design of soft robots requires accurate and stable simulations. In this article, we present an inverse finite element approach enabling simulation-ready characterization of soft robotic materials from uniaxial test data. Representing test specimens with finite elements, and modeling the specimen-device coupling, we enable the simultaneous fitting of hyperelastic parameters to single or multiple tests performed on specimens of varying shape and size. To safeguard against simulation instabilities, or non-physical behavior, we reparameterize and bound parameters using the consistency with linear elasticity. We use our characterization on commonly used silicones, and discuss the resolution- and order-dependence of fitted parameters, and the Mullins effect in the context of simulation-based design.

This article presents a system for soft human–robot handshaking, using a soft robot hand in conjunction with a lightweight and impedance-controlled robot arm. Using this system, we study how different factors influence the perceived naturalness, and give the robot different personality traits. Capitalizing on recent findings regarding handshake grasp force regulation, and on studies of the impedance control of the human arm, we investigate the role of arm stiffness as well as the kinesthetic synchronization of human and robot arm motions during the handshake. The system is implemented using a lightweight anthropomorphic arm, with a Pisa/IIT Softhand wearing a sensorized silicone glove as the end-effector. The robotic arm is impedance-controlled, and its stiffness changes according to different laws under investigation. An internal observer is employed to synchronize the human and robot arm motions. Thus, we simulate both active and passive behavior of the robotic arm during the interaction. Using the system, studies are conducted where 20 participants are asked to interact with the robot, and then rate the perceived quality of the interaction using Likert scales. Our results show that the control of the robotic arm kinesthetic behavior does have an effect on the interaction with the robot, in term of its perceived personality traits, responsiveness, and human-likeness. Our results pave the way towards robotic systems that are capable of performing human–robot interactions in a more human-like manner, and with personality.

We propose a novel generic shape optimization method for CAD models based on the eXtended Finite Element Method (XFEM). Our method works directly on the intersection between the model and a regular simulation grid, without the need to mesh or remesh, thus removing a bottleneck of classical shape optimization strategies. This is made possible by a novel hierarchical integration scheme that accurately integrates finite element quantities with sub-element precision. For optimization, we efficiently compute analytical shape derivatives of the entire framework, from model intersection to integration rule generation and XFEM simulation. Moreover, we describe a differentiable projection of shape parameters onto a constraint manifold spanned by user-specified shape preservation, consistency, and manufacturability constraints. We demonstrate the utility of our approach by optimizing mass distribution, strength-to-weight ratio, and inverse elastic shape design objectives directly on parameterized 3D CAD models.

We present a system for fast and robust handovers with a robot character, together with a user study investigating the effect of robot speed and reaction time on perceived interaction quality. The system can match and exceed human speeds and confirms that users prefer human-level timing. The system has the appearance of a robot character, with a bear-like head and a soft anthropomorphic hand and uses Bézier curves to achieve smooth minimum-jerk motions. Fast timing is enabled by low latency motion capture and real-time trajectory generation: the robot initially moves towards an expected handover location and the trajectory is updated on-the-fly to converge smoothly to the actual handover location. A hybrid automaton provides robustness to failure and unexpected human actions. In a 3x3 user study, we vary the speed of the robot and add variable sensorimotor delays. We evaluate the social perception of the robot using the Robot Social Attribute Scale (RoSAS). Inclusion of a small delay, mimicking the delay of the human sensorimotor system, leads to an improvement in perceived qualities over both no delay and long delay conditions. Specifically, with no delay the robot is perceived as more discomforting, and with a long delay it is perceived as less warm.

Facial animation is one of the most challenging problems in computer graphics, and it is often solved using linear heuristics like blend-shape rigging. More expressive approaches like physical simulation have emerged, but these methods are very difficult to tune, especially when simulating a real actorÕs face. We propose to use a simple finite element simulation approach for face animation, and present a novel method for recovering the required simulation parameters in order to best match a real actor's face motion. Our method involves reconstructing a very small number of head poses of the actor in 3D, where the head poses span different configurations of force directions due to gravity. Our algorithm can then automatically recover both the gravity- free rest shape of the face as well as the spatially-varying physical material stiffness such that a forward simulation will match the captured targets as closely as possible. As a result, our system can produce actor-specific, physical parameters that can be immediately used in recent physical simulation methods for faces. Furthermore, as the simulation results depend heavily on the chosen spatial layout of material clusters, we analyze and compare different spatial layouts.

Creating animations for robotic characters is very challenging due to the constraints imposed by their physical nature. In particular, the combination of fast motions and unavoidable structural deformations leads to mechanical oscillations that negatively affect their performances. Our goal is to automatically transfer motions created using traditional animation software to robotic characters while avoiding such artifacts. To this end, we develop an optimization-based, dynamics-aware motion retargeting system that adjusts an input motion such that visually salient low-frequency, large amplitude vibrations are suppressed. The technical core of our animation system consists of a differentiable dynamics simulator that provides constraint-based two-way coupling between rigid and flexible components. We demonstrate the efficacy of our method through experiments performed on a total of five robotic characters including a child-sized animatronic figure that features highly dynamic drumming and boxing motions.

Elastically deforming wire structures are lightweight, durable, and can be bent within minutes using CNC bending machines. We present a computational technique for the design of kinetic wire characters, tailored for fabrication on consumer-grade hardware. Our technique takes as input a network of curves or a skeletal animation, then estimates a cable-driven, compliant wire structure which matches user-selected targets or keyframes as closely as possible. To enable large localized deformations, we shape wire into functional spring-like entities at a discrete set of locations. We first detect regions where changes to local stiffness properties are needed, then insert bendable entities of varying shape and size. To avoid a discrete optimization, we first optimize stiffness properties of generic, non-fabricable entities which capture well the behavior of our bendable designs. To co-optimize stiffness properties and cable forces, we formulate an equilibrium-constrained minimization problem, safeguarding against inelastic deformations. We demonstrate our method on six fabricated examples, showcasing rich behavior including large deformations and complex, spatial motion.

Large-scale binder jetting provides a promising alternative to manual sculpting of sandstone. The weak build material, however, severely limits its use in architectural ornamentation. We propose a structural optimization that jointly optimizes an ornament's strength-to-weight ratio and balance under self-weight, thermal, wind, and live loads. To account for the difference in the tensile and compressive strength of the build material, we turn the Bresler-Pister criterion into a failure potential, measuring the distance to failure. Integrated into an XFEM-based level set formulation, we minimize this potential by changing the topology and shape of the internal structure. To deal with uncertainties in the location of live loads, and the direction of wind loads, we first estimate loads that lead to the weakest structure, then minimize the potential of failure under identified worst-case loads. With the help of first-order optimality constraints, we unify our worst-case load estimation and structural optimization into a continuous optimization. We demonstrate applications in art, furniture design, and architectural ornamentation with three large-scale 3D printed examples.

Facial performance capture is the primary method for generating facial animation in video games, feature films and virtual environments, and recent advances have produced very compelling results. Still, one of the most challenging regions is the mouth, which often contains systematic errors due to the complex appearance and occlusion/dis-occlusion of the lips. We present a novel user-guided approach to correcting these common lip shape errors present in traditional capture systems. Our approach is to allow a user to manually correct a small number of problematic frames, and then our system learns the types of corrections desired and automatically corrects the entire performance. As correcting even a single frame using traditional 3D sculpting tools can be time consuming and require great skill, we also propose a simple and fast 2D sketch-based method for generating plausible lip corrections for the problematic key frames. We demonstrate our results on captured performances of three different subjects, and validate our method with an additional sequence that contains ground truth lip reconstructions.

We present a method for designing and fabricating MetaSilicones — composite silicone rubbers that exhibit desired macroscopic mechanical properties. The underlying principle of our approach is to inject spherical inclusions of a liquid dopant material into a silicone matrix material. By varying the number, size, and locations of these inclusions as well as their material, a broad range of mechanical properties can be achieved. The technical core of our approach is formed by an optimization algorithm that, combining a simulation model based on extended finite elements (XFEM) and sensitivity analysis, computes inclusion distributions that lead to desired stiffness properties on the macroscopic level. We explore the design space of MetaSilicone on an extensive set of simulation experiments involving materials with optimized uni- and bi-directional stiffness, spatially-graded properties, as well as multi-material composites. We present validation through standard measurements on physical prototypes, which we fabricate on a modified filament-based 3D printer, thus combining the advantages of digital fabrication with the mechanical performance of silicone elastomers.

Handshakes are common greetings, and humans therefore have strong priors of what a handshake should feel like. This makes it challenging to create compelling and realistic human-robot handshakes, necessitating the consideration of human haptic perception in the design of robot hands. At its most basic level, haptic perception is encoded by contact points and contact pressure distributions on the skin. This motivates our work on measuring the contact area and contact pressure in human handshaking interactions. We present two benchmarking experiments in this regard, measuring the contact locations in human-human/human-robot handshaking and the contact pressure distribution for handshakes with a sensorized palm. We present results from human studies with the benchmarking experiments, providing a baseline for comparison with robot hands as well as presenting new insights into human handshaking. We also show initial work in using these results for the evaluation of robot hands, and progressing towards iterative design of robot hands optimized for social hand interactions.

In this paper, we present an optimization-based approach for the design of cable-driven kinematic chains and trees. Our system takes as input a hierarchical assembly consisting of rigid links jointed together with hinges. The user also specifies a set of target poses or keyframes using inverse kinematics. Our approach places torsional springs at the joints and computes a cable network that allows us to reproduce the specified target poses. We start with a large set of cables that have randomly chosen routing points and we gradually remove the redundancy. Then we refine the routing points taking into account the path between poses or keyframes in order to further reduce the number of cables and minimize required control forces. We propose a reduced coordinate formulation that links control forces to joint angles and routing points, enabling the co-optimization of a cable network together with the required actuation forces. We demonstrate the efficacy of our technique by designing and fabricating a cable-driven, animated character, an animatronic hand, and a specialized gripper.

Spinning tops and yo-yos have long fascinated cultures around the world with their unexpected, graceful motions that seemingly elude gravity. We present an algorithm to generate designs for spinning objects by optimizing rotational dynamics properties. As input, the user provides a solid 3D model and a desired axis of rotation. Our approach then modifies the mass distribution such that the principal directions of the moment of inertia align with the target rotation frame. We augment the model by creating voids inside its volume, with interior fill represented by an adaptive multi-resolution voxelization. The discrete voxel fill values are optimized using a continuous, nonlinear formulation. Further, we optimize for rotational stability by maximizing the dominant principal moment. We extend our technique to incorporate deformation and multiple materials for cases where internal voids alone are insufficient. Our method is well-suited for a variety of 3D printed models, ranging from characters to abstract shapes. We demonstrate tops and yo-yos that spin surprisingly stably despite their asymmetric appearance.

We present a computational tool for designing compliant mechanisms. Our method takes as input a conventional, rigidly-articulated mechanism defining the topology of the compliant design. This input can be both planar or spatial, and we support a number of common joint types which, whenever possible, are automatically replaced with parameterized flexures. As the technical core of our approach, we describe a number of objectives that shape the design space in a meaningful way, including trajectory matching, collision avoidance, lateral stability, resilience to failure, and minimizing motor torque. Optimal designs in this space are obtained as solutions to an equilibrium-constrained minimization problem that we solve using a variant of sensitivity analysis. We demonstrate our method on a set of examples that range from simple four-bar linkages to full-fledged animatronics, and verify the feasibility of our designs by manufacturing physical prototypes.

Soft robotic actuators use compliance to passively adapt their shape to objects and the environment. The specific choices of design parameters decide the behavior of the actuator during these interactions. By analyzing human-human hand interactions we observed that the distribution of contact pressure is a relevant benchmarking metric to judge “humanness” and comfort of the interaction. In order to understand how the choice of design parameters of soft actuators influences the benchmarking results, we created a set of prototypes with different design parameters. By measuring and comparing the contact pressure distributions we can reason about which design properties are most relevant for more human-like interactions.

Oftentimes facial animation is created separately from overall body motion. Since convincing facial animation is challenging enough in itself, artists tend to create and edit the face motion in isolation. Or if the face animation is derived from motion capture, this is typically performed in a mo-cap booth while sitting relatively still. In either case, recombining the isolated face animation with body and head motion is non-trivial and often results in an uncanny result if the body dynamics are not properly reflected on the face (e.g. the bouncing of facial tissue when running). We tackle this problem by introducing a simple and intuitive system that allows to add physics to facial blendshape animation. Unlike previous methods that try to add physics to face rigs, our method preserves the original facial animation as closely as possible. To this end, we present a novel simulation framework that uses the original animation as per-frame rest-poses without adding spurious forces. As a result, in the absence of any external forces or rigid head motion, the facial performance will exactly match the artist-created blendshape animation. In addition, we propose the concept of blendmaterials to give artists an intuitive means to account for changing material properties due to muscle activation. This system allows to automatically combine facial animation and head motion such that they are consistent while preserving the original animation as closely as possible. The system is easy to use and readily integrates with existing animation pipelines.

Soft robotic technologies are paving way for physical human-robot hand interactions, creating a need for structured evaluation metrics for robot hands. We propose that the contact pressure distribution of the grasp should be used as a hand benchmark both for naturalness and comfort, and present our initial work in this direction. We describe an experimental setup for measuring the contact pressure distribution and present a case study comparing the pressure distributions from a robotic hand and a human hand. The grasping force of the human hand is ten times greater than the robot, but the robot hand produces higher peak contact pressures and smaller contact areas.

We present an algorithm to balance 3D printed models using movable embedded masses. As input, the user provides a 3D model together with the desired suspension, standing, and immersion objectives. Our technique then determines the placement and suitable sizing of a set of hollow capsules with embedded metallic spheres, leveraging the resulting multiple centers of mass to simultaneously satisfy the combination of these objectives. To navigate the non-convex design space in a scalable manner, we propose a heuristic that leads to near-optimal solutions when compared to an exhaustive search. Our method enables the design of models with complex and surprising balancing behavior, as we demonstrate with several manufactured examples.

We present a novel optimization-based algorithm for the design and fabrication of customized, deformable input devices, capable of continuously sensing their deformation. We propose to embed piezoresistive sensing elements into flexible 3D printed objects. These sensing elements are then utilized to recover rich and natural user interactions at runtime. Designing such objects is a challenging and hard problem if attempted manually for all but the simplest geometries and deformations. Our method simultaneously optimizes the internal routing of the sensing elements and computes a mapping from low-level sensor readings to user-specified outputs in order to minimize reconstruction error. We demonstrate the power and flexibility of the approach by designing and fabricating a set of flexible input devices. Our results indicate that the optimization-based design greatly outperforms manual routings in terms of reconstruction accuracy and thus interaction fidelity.

We present a method for interactive editing of planar linkages. Given a working linkage as input, the user can make targeted edits to the shape or motion of selected parts while preserving other, e.g., functionally-important aspects. In order to make this process intuitive and efficient, we provide a number of editing tools at different levels of abstraction. For instance, the user can directly change the structure of a linkage by displacing joints, edit the motion of selected points on the linkage, or impose limits on the size of its enclosure. Our method safeguards against degenerate configurations during these edits, thus ensuring the correct functioning of the mechanism at all times. Linkage editing poses strict requirements on performance that standard approaches fail to provide. In order to enable interactive and robust editing, we build on a symbolic kinematics approach that uses closed-form expressions instead of numerical methods to compute the motion of a linkage and its derivatives. We demonstrate our system on a diverse set of examples, illustrating the potential to adapt and personalize the structure and motion of existing linkages. To validate the feasibility of our edited designs, we fabricated two physical prototypes.

Spinning tops and yo-yos have long fascinated cultures around the world with their unexpected, graceful motions that seemingly elude gravity. We present an algorithm to generate designs for spinning objects by optimizing rotational dynamics properties. As input, the user provides a solid 3D model and a desired axis of rotation. Our approach then modifies the mass distribution such that the principal directions of the moment of inertia align with the target rotation frame. We augment the model by creating voids inside its volume, with interior fill represented by an adaptive multi-resolution voxelization. The discrete voxel fill values are optimized using a continuous, nonlinear formulation. Further, we optimize for rotational stability by maximizing the dominant principal moment. We extend our technique to incorporate deformation and multiple materials for cases where internal voids alone are insufficient. Our method is well-suited for a variety of 3D printed models, ranging from characters to abstract shapes. We demonstrate tops and yo-yos that spin surprisingly stably despite their asymmetric appearance.

The desktop publishing revolution of the 1980s is currently repeating itself in 3D, referred to as desktop manufacturing. Online services such as Shapeways have become available, making personalized manufacturing on cutting edge additive manufacturing (AM) technologies accessible to a broad audience. Affordable desktop printers will soon take over, enabling people to fabricate custom 3D models at home. Contemporary AM technologies have advanced enough to enable 3D printing at high resolution, in full-color, and with mixtures of soft and hard materials. As opposed to subtractive manufacturing (SM) such as milling or drilling, they can fabricate highly complex assemblies without the need for a manual assembly of individual components. Yet, one of the major issues holding back widespread use of AM is the lack of efficient algorithms for the automated fabrication of digital CG, and the reproduction of physical content. Besides, we do not have tools at our disposal that aid us with the design of multi-material content or complex assembly structures. For physical reproduction, we strive for methods to acquire properties such as, e.g., reflectance (appearance) or elasticity (deformation behavior) from real-world objects, representing them digitally, then automating their fabrication using AM. However, the vast majority of digital 3D content are directly designed on computers, hence, potentially exhibit a highly non-physical behavior. To fabricate such content, we seek methods for the automated estimation of physical models from these digital ones. This dissertation examines computational aspects of 3D manufacturing. In particular, we investigate design tools and automated fabrication of an object’s deformation behavior, articulation, and geometry. We present a complete process for measuring, representing, simulating, and physically fabricating an object’s elastic deformation behavior. This process enables the reproduction of physical deformation behavior. Furthermore, we introduce a technique for the automated fabrication of articulated models, estimated from the most widely used format in character animation – so called skinned meshes. Our technique estimates assemblies, approximating this inherently non-physical input in a piecewise linear manner. Lastly, we propose a method for the scale-aware fabrication of static geometry, capable of abstracting, then engraving details that cannot be fabricated on a pre-specified 3D printer.

In this paper, we propose a system to determine the pressure gradient at rest in the aorta. We developed a technique to efficiently initialize a regular simulation grid from a patient-specific aortic triangulated model. On this grid we employ the lattice Boltzmann method to resolve the characteristic fluid flow through the vessel. The inflow rates, as measured physiologically, are imposed providing accurate pulsatile flow. The simulation required a resolution of at least 20 microns to ensure a convergence of the pressure calculation. HARVEY, a large-scale parallel code, was run on the IBM Blue Gene/Q supercomputer to model the flow at this high resolution. We analyze and evaluate the strengths and weaknesses of our system.

We introduce a mixed-initiative approach for document reconstruction that can significantly reduce the amount of time and effort required to reassemble a document from shredded pieces or an artifact from broken fragments. We focus in particular on the hardest subproblem, which is the problem of identifying a matching neighbor for any given piece. Our approach, called hallucination, combines human and machine intelligence by leveraging people’s ability to draw what a neighboring piece may look like, and then using the drawing as a template based on which the computer computes likely matches. Experiments on a puzzle from the DARPA Shredder Challenge demonstrate that the hallucination approach significantly reduces the search space for identifying a match, outperforming humans and computers working in isolation.

Articulated deformable characters are widespread in computer animation. Unfortunately, we lack methods for their automatic fabrication using modern additive manufacturing (AM) technologies. We propose a method that takes a skinned mesh as input, then estimates a fabricatable single-material model that approximates the 3D kinematics of the corresponding virtual articulated character in a piecewise linear manner. We first extract a set of potential joint locations. From this set, together with optional, user-specified range constraints, we then estimate mechanical friction joints that satisfy inter-joint non-penetration and other fabrication constraints. To avoid brittle joint designs, we place joint centers on an approximate medial axis representation of the input geometry, and maximize each joint’s minimal cross-sectional area. We provide several demonstrations, manufactured as single, assembled pieces using 3D printers.

This paper introduces a data-driven process for designing and fabricating materials with desired deformation behavior. Our process starts with measuring deformation properties of base materials. For each base material we acquire a set of example deformations, and we represent the material as a non-linear stress-strain relationship in a finite-element model. We have validated our material measurement process by comparing simulations of arbitrary stacks of base materials with measured deformations of fabricated material stacks. After material measurement, our process continues with designing stacked layers of base materials. We introduce an optimization process that finds the best combination of stacked layers that meets a user’s criteria specified by example deformations. Our algorithm employs a number of strategies to prune poor solutions from the combinatorial search space. We demonstrate the complete process by designing and fabricating objects with complex heterogeneous materials using modern multi-material 3D printers.

This paper introduces a data-driven representation and modeling technique for simulating non-linear heterogeneous soft tissue. It simplifies the construction of convincing deformable models by avoiding complex selection and tuning of physical material parameters, yet retaining the richness of non-linear heterogeneous behavior. We acquire a set of example deformations of a real object, and represent each of them as a spatially varying stress-strain relationship in a finite-element model. We then model the material by non-linear interpolation of these stress-strain relationships in strain-space. Our method relies on a simple-to-build capture system and an efficient run-time simulation algorithm based on incremental loading, making it suitable for interactive computer graphics applications. We present the results of our approach for several non-linear materials and biological soft tissue, with accurate agreement of our model to the measured data.

In this project, I describe an image processing framework that uses locally weighted least squares regression to denoise, reconstruct and upsample images. Classic, bilateral and robust kernel regression is derived and discussed.

In this thesis, a novel representation and technique for simulating static non-linear material behavior based on Finite Elements (FE) is presented. All required simulation parameters can be acquired and fitted from a set of example deformations of a real-world object or subject. The simulation is therefore closely related to the person or object specific deformation behavior. We first acquire a single static surface scan and several measurements of static surface displacements by probing an object at many positions and orientations using a force sensor. A trinocular stereo system measures the surface displacements at colored marker locations on the object. The volume of the object is discretized into tetrahedral elements, and for each element and every measurement material parameters are fitted. Our material model consists of material parameters and the corresponding material strain. During run time, we blend these parameters by using a novel strain-based interpolation scheme in material strain space, modeling therefore intuitively the non-linear material stress-strain relationship. Furthermore, since the model is based on a linear deformation FEM, simulations of new interactions are stable and also computationally efficient.

The method of Moving Least Squares (MLS) is a popular framework for reconstructing continuous functions from scattered data due to its rich mathematical properties and well-understood theoretical foundations. This paper applies MLS to volume rendering, providing a unified mathematical framework for ray casting of scalar data stored over regular as well as irregular grids. We use the MLS reconstruction to render smooth isosurfaces and to compute accurate derivatives for high-quality shading effects. We also present a novel, adaptive preintegration scheme to improve the efficiency of the ray casting algorithm by reducing the overall number of function evaluations, and an efficient implementation of our framework exploiting modern graphics hardware. The resulting system enables high-quality volume integration and shaded isosurface rendering for regular and irregular volume data.