The present disclosure presents a visuo-haptic sensor, which is based on a passive, deformable element whose deformation is observed by a camera. In particular, the improved simplified sensor may determine the force and/or torque applied to a point of the sensor in multiple spatial dimensions.
This application is a continuation of U.S. patent application Ser. No. 15/592,326, now U.S. Pat. No. 10,393,603, and claims the benefit of German patent application No. 10 2016 108 966.4, filed 13 May 2016, which is incorporated herein by reference.TECHNICAL FIELD
The present disclosure presents a visuo-haptic sensor, which is based on a passive, deformable element whose deformation is observed by a camera. In particular, an improved simplified sensor may determine the force and/or torque applied to a point of the sensor in multiple spatial dimensions.BACKGROUND
Autonomous robots rely on a variety of different sensor systems to obtain a rich, multi-modal representation of the environment. Haptic perception mainly depends on force sensors, which are available in many variants.
Force sensors either measure a force, a torque, or both force and torque (force and/or torque) vector at a single point, or a contact force distributed over a surface.
Force sensors that measure contact force profiles are often categorized as tactile sensors. In general, forces are kinesthetic data and not tactile data. Thus, the distinction between a force sensor and a tactile sensor is often not clear, which is why we use the generic term “haptic sensor”.
Force and/or torque sensors are usually installed between a robot arm and a tool (the end-effector) in order to measure the contact forces between a tool and the environment at a single point during grasping, manipulation tasks like mounting or placing of objects as well as gravitational and acceleration forces while moving objects.
Modern robots are also equipped with visual sensors, such as cameras. Cameras in the visual spectrum are the basic sensor for most computer vision algorithms such as object detection, visual search or pose estimation. An exemplary visuo-haptic sensor is disclosed in U.S. patent application publication 2016/0107316, published 21 Apr. 2016, the complete disclosure of which is herein incorporated by reference.SUMMARY
According to a first aspect of the present disclosure, an apparatus 1 is provided comprising a haptic element comprising at least one passive elastically deformable element 3 which is deformable in multiple dimensions. At least one camera 4 external to the haptic element and operable to capture images of the elastically deformable element 3 is provided. Further, a processor 5 is also provided, wherein the processor 5 is operable to determine a change of pose of the elastically deformable element 3 and determine a measurement of force and torque 10 applied to the elastically deformable element 3 based on the captured images and the determined change of pose, wherein the measurement includes at least three components comprising forces and/or torques. This enables measurement of a force and/or torque 10 reading at a single point in multiple spatial dimensions i.e. more than two dimensions using passive elements, and obviates the need for complex integrated circuitry to be built into the haptic element.
According to one embodiment thereof, the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions. The present disclosure advantageously makes it possible to determine a full 6D force and/or torque measurement using the passive elements, thereby providing a high level of accuracy due to achieving a measurement including the maximum number of dimensions for both force and torque.
According to a further embodiment, the haptic element further comprises a rigid base element 6 and a rigid front element 7 coupled via the elastically deformable element 3. Additionally, according to this embodiment, the processor 5 is further operable to determine the change of pose of the rigid front element with respect to the base element.
According to one embodiment thereof, the deformable element consists of two parts, a spring 31 and a beam 32 mounted between the base element 6 and the front element 7 and connected at point 38. In order to determine the multiple spatial measurement of force, forces and torques applied at tool 9 along or around axes Y, Z, lead to a deflection of the beam as shown in
According to a different embodiment thereof, the rigid base element 6 and rigid front element 7 comprise planar structures 80, 81. By providing planar structures, a larger surface area (than a beam) may be created which facilitates determination of minimal changes in pose of the elements being captured. In this way, the force detection sensitivity of the apparatus 1 is increased. In a preferred exemplary implementation described below, a disk 61 implements both the base element and one planar structure, and a disk 71 implements both the front element and the second planar structure. The skilled person will appreciate that many other shapes of planar structure such as H or T shapes would be suitable. In some cases, the rigid front element 7 may have a smaller surface area with respect to the rigid base element 6. This provides a convenient means for expressing the pose of the rigid front element 7 within the coordinate frame of rigid base element 6.
In an embodiment of the first aspect, the change of pose is determined by observing at least two points on the haptic element. This is possible when using a depth and a depth gradient from a depth camera as the camera 4 observing the haptic element. According to this embodiment, a simpler technique than tracking templates or patterns is provided for determining force. An advantage of this embodiment is that less data processing is required, thereby making minimal impact on valuable processing resources needed for other parts of a system in which the apparatus 1 may be integrated.
In a further embodiment, the change of pose is determined by observing three or more points on the haptic element. This is possible when using a normal camera 4 to observe the haptic element. The increased number of observation points leads to increased accuracy of the force and/or torque measurement.
In yet another embodiment, the change of pose is determined by a visual tracker. Visual tracking allows for highly accurate pose estimation in comparison to many other known methods which provide results with such a level of inaccuracy to not be feasible.
In another preferred embodiment, the haptic element is mechanically attached to a movable robotic element and the movable robotic element is included in the field of view of the camera 4. This advantageously provides a more efficient use of resources as the camera 4 can also be used for other applications e.g. tracking movement of a robotic arm to which the sensor is attached.
In a further preferred embodiment, the haptic element is mounted within the structure of a robotic arm, manipulator or a kinematic chain. By virtue of using passive elements, the present disclosure facilitates integration with the structure since it does not create an additional discrete electronic system which could otherwise undesirably interfere with the distinct electronic systems of the e.g. robotic arm.
In a second aspect of the present disclosure, a method is provided, the method comprising the steps of providing a haptic element comprising at least one passive elastically deformable element 3; capturing images of the haptic element with at least one camera 4; determining a change of pose of the passive elastically deformable element 3; and determining a measurement of force and torque 10 applied to the passive elastically deformable element 3 based on the captured images and the determined change of pose, wherein the measurement includes at least three components comprising forces and/or torques.
According to a preferred embodiment of the second aspect, the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions.
According to further embodiments of the second aspect, the method further comprises a step of determining a change in pose of elastically deformable element by at least one of: observing at least two points on the haptic element, by observing three or more points on the haptic element and/or visually tracking the haptic element.
According to a preferred exemplary implementation, a visuo-haptic sensor is presented, which uses standard cameras 41,42 to obtain haptic data, i.e. force and torque 10 during manipulation operations. The camera 4 measures a haptic element, consisting of deformable element 3, which is mounted between a robot actuator (such as a robotic arm) and a tool, such as a gripper, a screwdriver or a sensor tip. The deformation of this element is converted to a multidimensional force and/or torque reading based on its known deformation model. Ideally, a 6D force and/or torque 10 applied to the tool 9 is determined from the deformation, which is calculated from the camera image. According to this example, deformable elements 3 may be made of plastic or rubber, as well as with a beam-like structure. However, the skilled person will appreciate that Euler-Bernoulli beam theory is equally applicable to complex cross-sections, such as those comprising an H or T shape. Visual observations from the object, the scene or the components of the robot may be acquired by the same camera 4 and are thus naturally coherent with haptic data. This is an important advantage over existing haptic sensors with optical readout, which rely on an optical sensor inside of a deformable structure. Dedicated sensor systems can be replaced by a low-cost camera 4, which may be required for other tasks already, reducing costs and system complexity. Integration of visual data is important to observe the reaction of an object during manipulation, and also to verify the state of the manipulator itself. The accuracy of the sensor is shown to be good by comparison with an industrial force sensor. According to preferred implementations, two main embodiments are realized. The first one uses templates to observe the deformation of a deformable element comprising a spring 31 and a metallic beam 32 whilst the second one observes two disks connected by one or several deformable elements 33. This design allows for low-cost force and/or torque sensors which are naturally coherent with the visual modality as stated above.Advantages
Compared to existing tactile/haptic sensors, the present disclosure offers a number of benefits:
Coherent visual and haptic/tactile measurements: The proposed sensor extracts haptic information from the camera image. Visual and haptic measurements are therefore naturally coherent, i.e. they are sampled and made available at the same point in time.
Smooth transition between haptic and visual data: Since both modalities are derived from the same data (i.e. the image), intermediate representation can be obtained. With separate sensor systems, their data representations cannot be easily converted. A hard switch between them might be required.
Mitigation of shortcomings of a single modality: For the same reasons as discussed above, missing data from one modality can be replaced by the other one. For instance, if the pose of a transparent object cannot be exactly determined based on visual methods, a refinement is possible based on haptic information.
Natural compliance, i.e. deformable elements result in “softness” of end-effectors. Compliance is an important safety feature for robots working together with humans.
Reduced effort for calibration: Mutual calibration of different sensors is a tedious task. With the present disclosure, calibration of cameras is sufficient.
Reduced system complexity: Dedicated sensor modules have their own processing units, their own firmware and require interfacing to a central processing unit. This results in a significant overhead during production, maintenance and calibration, since sensor units must be mounted all around a robot. Our system just on central processing and just requires passive mechanic components around the robot.
Reduced effort for cabling/networking: The passive deformable elements 3 do not require any cabling. They can thus be easily placed on any part of the robot with.
Very low costs: A new sensitive element is completely passive and can be produced for less than 1 EUR. Hereby, it is assumed that a vision system (camera 4, vision processor 5) is already available to observe the passive element(s) 3.
Any disadvantages according to the present disclosure are limited to low-end systems without any powerful processing unit wherein the addition of cameras and additional computers would not be feasible.
The present disclosure is particularly suited for complex robots which rely on multiple visual sensors and powerful processing units and can therefore profit the most from the aforementioned advantages.
Compared to purely vision-based sensors, the present disclosure acknowledges the need for haptic sensors which are essential for all kinds of manipulation tasks, even if highly accurate vision sensors are present. This applies both to existing haptic sensor systems and to the present disclosure.
A combined tracker/detector for templates on planar structures 80,81,82 is used to find and follow a texture with an image or video provided by a camera 4. It is based on scale-invariant feature transform (SIFT) for feature detection and visual template tracking such as efficient second-order minimization (ESM) for tracking and refinement. Thus, the exact 6D pose of objects can be determined in realtime, as long as a model of the templates, i.e., an image with scale information, is available. The scale is determined from the dpi (dots-per-inch) value of the image file. The system detects/tracks multiple arbitrary planar templates such as printed designs or natural photos, simultaneously. Each template model is initialized on startup, i.e. SIFT features are extracted, and the tracker performs pre-calculations. Many artificial objects can be tracked directly using their printed texture as a model. Also, it is straight-forward to create real objects from template models at the exact scale with a standard printer. These templates are easily attached anywhere on a robot or an object. They offer more flexibility, ease of use and a higher tracking quality than 3D models or the pre-defined markers from the open source computer tracking library ARtoolkit.
The system relies on a GPU implementation of SIFT to search for the template models in the image. This step is too slow for processing at a framerate of 30 Hz on mid-range GPUs and should be avoided, if possible. Tracking is performed with ESM, which is either initialized from SIFT or from the pose of the previous frame. The latter case is preferred, since it allows skipping SIFT detection. Poses from ESM are much more accurate and less noisy than from SIFT matches. Tracking is always done with the “original” model, such that there is no drift, even if performed over long periods of time. Processing is split in multiple threads, running at different rates. Thread A performs SIFT detection and matching, if there is currently any template whose pose is unknown. Matching provides an estimate of the homography. The ESM runs in thread B at framerate and tracks all templates, for which initial poses are available. If required, each tracker may run in a separate thread to reduce latency on multi-core CPUs.
The real-world poses of the templates with respect to the camera 4 is obtained by homography decomposition, given the camera parameters.
Modeling deformation of beams
The deformation of beams or rods under load is described by the Euler-Bernoulli beam theory for small deflections. Here, a force and/or moment 10 perpendicular to the beam is applied at its front element 7, see
The other end of the beam 32 is considered to be fixed (“clamped”). In principle, the beam can be made of elastic material such as spring steel or plastic. Yet, no plastic (i.e. permanent) deformations should occur within the relevant range of deformations. We use a beam 32 made of spring steel with a diameter of about 2 mm and a length of 10-20 cm.
The deflection curve w(x) in 2D describes the deformation of the beam 32 along the y-axis over the x-axis:
This is the commonly-used approximation for small deflection, which ignores the shortening of the deformed beam 32 along the x-axis. The values for the elastic modulus E and the second moment of area I are constants for uniform and homogeneous beams. For a circular cross section of the beam 32 with radius r.
There is no distributed load, such that q(x)=0. Quadruple integration of (1) yields four integration constants, which are used to fulfil the boundary conditions. Due to the clamping at x=0, w(0)=w′(0)=0. The derivatives of w have a distinct physical meaning—specifically the moment is M=−EIw″, and the shear force is Q=−EIw′. Therefore, when a force F 10 is applied perpendicular to the beam at its end, boundary conditions are:
The force 10 is applied at x=L, where L is the length of the beam 32. With four boundary conditions, a unique solution wF can be given for Eqn. (1). Similarly, for a moment or torque M applied at point x=L, a solution wM is determined using the boundary conditions:
Since the differential equation is linear, the two solutions can be superimposed. For a force and moment applied at ξ=1, with
and clamping at x=0, we obtain:
The deflection of the real beam is observed by a camera 4 at one or multiple points of the curve w. Extension of the discussed 2D case to 3D is straight-forward by separation along y and z, yielding deformation curves wy, wz, see
If multiple observations are available, we obtain an overdetermined system [w|ξ
A photo based on this structure is shown in
Observations of the deflection curve of the beam w may be obtained in different ways: An edge tracker may be used to track the two contours along the beam and provide a dense sampling of w. This approach provides a high robustness to occlusion, due to the large number of tracked points. However, point tracking does not provide any reliable depth information, basically limiting force measurements to 1D. Simple features along the beam 32—such as colored spheres—would allow for easier detection, yet still with very limited depth resolution. A template tracker/detector, see the above background section describing a combined detector and tracker, on the other hand, provides a full 6D pose, including accurate depth values and the full rotation, which corresponds to the derivatives of w′Y,Z. Planar textured templates are attached along the beam 32 at a single fixture point via planar structures 80,81 respectively for this tracker. They may be integrated into the design of the robot case. The pose of the beam 32 at the fixture point and the template are related by a constant rigid transformation determined by the mounting structure. Only a low number of templates can be attached along the rod, resulting in a limited number of observations of w and w′. The acting force/moment 10 is calculated from these observations using (3) as outlined there.
As indicated in
Like that, the pose of the sensor may vary, and the camera 4 can move relative to the sensor. In principle, the reference frame may be fixed to the camera frame or calculated from a robot model. Yet, errors can be reduced significantly, if a reference frame is obtained within the image, close to where measurements are performed. Two templates are attached along the beam 32, at fixed relative positions Since the templates on 81,82 are not aligned perfectly, and the beam 32 might exhibit some plastic deformation, a zeroing procedure is performed at startup, with zero force/moment 10 applied to the sensor. The corresponding resting poses TAU,BCn are expressed relative to the frame
Measurements of the template poses are performed for each video frame and expressed relative to their (constant) resting poses, which in turn are based on the current (variable) reference frame
6D Force and/or Torque Sensor with Unconstrained Beam/Spring
Another embodiment of the 6D force and/or torque sensor based on two passive elastic elements 31,32 is built according to
As shown in
Templates or any other kind of visual feature that allows for visual 6D pose estimation are placed on planar structures 80,81,82 attached at or near 6, 38 and 7. The 6D poses (position and rotation) of these locations are known from the feature pose and the constant offset between the location and its respective feature. Accurate 6D poses can be obtained, for instance, with a combined tracker/detector, as described above. Placing the features close to the given locations ensures a high pose accuracy. Additional features may be placed anywhere along the flexible elements to improve the accuracy of pose estimation. A camera is placed such that it observes all these features with an image resolution sufficient for the desired force and/or torque resolution. It may be fixed to the base 6, as for camera 42, or fixed to another structure, as for camera 41. In the former case, if the connection between 42 and 6 is known, feature 83 may be omitted. Otherwise, 6 serves as a local reference frame, which the other poses are expressed in.
The deformations of 31 and 32 are determined individually from the pose change between 6 and 38, or 38 and 7, respectively. Applied forces and/or torques 10 in 6D are determined using a deformation model, as outlined above. Compression of the spring 31 along its major axis caused by FX, as well as torsion around this axis caused by TX may be modeled according to Hooke's law. Deformation caused by FY,Z and TY,Z is modeled by the beam theory, applied individually for both elastic elements 31 and 32. In that case, two measurements are available, which improves measurement accuracy or robustness to partial occlusions. Deformation models may also be calibrated/learned using a generic model by applying test forces and/or torques to the sensor.
6D Force and/or Torque Sensor with Unconstrained Spring
In another embodiment, see
The two disks are made of a rigid material and serve as mounting plates. They can be of arbitrary shapes, whereby planar structures are most feasible. The disks are connected by one or several elastic elements 33, such as springs, rods of rubber or rubber foam. The elements should exhibit an elastic deformation behavior. The dimensions, arrangement and stiffness of these elements are chosen according to the required 6D sensitivity and measurement range of the sensor. There is a direct relationship between pose changes (in 6D) and a 6D force and/or torque 10 applied to 71, which is referred to as deformation model. This model can be obtained analytically, by simulation or by calibration. In the latter case, known force and/or torque values are applied, and the observed deformation is stored.
In case of a single beam-like deformable element, the relationship is expressed as discussed in the above background section describing modeling deformation of beams. The 6D poses of disks 61 and 71 are determined by at least one camera, such as cameras 41,42, based on features, templates or patterns on the disks 61,71. For instance, the pose of an arbitrary texture may be found as described in the above background section describing a combined detector and tracker. An even simpler approach is to find at least four uniquely identifiable features on each disk. If a depth camera is used, the poses can also be derived from the 3D surfaces or structure. A sufficient part of both disk must be observable for pose estimation.
In case of a camera position similar to camera 41, disk 71 could be made smaller, or it could be made partly transparent. In the latter case, parts of disk 61 that are seen through 71 are refracted. Displacements caused by refraction must be compensated. In case of camera arrangement according to camera 42, the inner surfaces of both disks 61,71 can be observed, except for parts that are occluded by the deformable elements 33. The pose of 71 is always expressed in the coordinate frame of 61, which allows both the cameras 41,42 and the entire sensor to move. A reference or zero pose is obtained when no external force 10 is applied to 71 or the tool 9. When an external force or torque 10 is applied, the pose of 71 changes from this reference pose. This change is converted to a force and/or torque based on the above-mentioned deformation model.
In preferred implementations, the sensor is built according to
The accuracy of the beam-based sensor is analyzed in an experiment with the sensor depicted in
The obtained results are shown in
Opening a crown cap, as used on many beverage bottles, is a good example for a manipulation operation which relies on joint visuo-haptic perception. For this experiment, a standard bottle opener is mounted onto a beam-based sensor according to
Bottles are recognized and tracked using the texture on their crown caps with a template-based detector/tracker. As soon as the user selects the desired bottle, the robot scans for it within its working space. Once the object has been detected and localized, the arm moves the tool 9 above the object and rotates it as required for the uncapping operation. The accuracy of this rotation movement is limited by two factors: Since the wrist joint of the arm is relatively far from the tool 9, a large motion is required in the joint space. This goes along with a larger positioning error (which could, however, be compensated by visual servoing). Yet, the used 2D tracker has only a limited depth accuracy—the height of the crown cap cannot be determined exactly. The alignment of the tool 9 on the crown cap must be very accurate in all dimensions in order to perform the uncapping successfully.
This level of precision is impossible using pure vision, especially for the depth direction. The exact position of the opener is determined haptically, by measuring the forces or moments with the visuo-haptic sensor. Refinement is first performed along the z-axis (height), the x-axis and finally the y-axis. Once the tool 9 is aligned, the arm rotates the tool 9 around its center point. If the crown cap has been removed successfully, the tracker/detector will no longer detect its texture on top of the bottle.
1. An apparatus, comprising:
- a haptic element comprising at least one passive elastically deformable element deformable in multiple dimensions;
- at least one camera external to the haptic element and operable to capture images of the elastically deformable element; and
- a processor operable to determine a change of pose of the elastically deformable element by visual tracking and determine a measurement of force and torque applied to the elastically deformable element based on the captured images and the determined change of pose, wherein the measurement includes at least three components comprising forces and/or torques.
2. The apparatus of claim 1 wherein the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions.
3. The apparatus of claim 1,
- wherein the haptic element further comprises a rigid base element and a rigid front element coupled via the elastically deformable element; and
- wherein the processor is further operable to determine the change of pose of the rigid front element with respect to the base element.
4. The apparatus of claim 3, wherein the deformable element comprises a beam and/or a tool which is mounted to the front element.
5. The apparatus of claim 3, wherein the rigid base element and rigid front element comprise planar structures.
6. The apparatus of claim 1, wherein the change of pose is determined by observing at least two points on the haptic element.
7. The apparatus of claim 1, wherein the change of pose is determined by observing three or more points on the haptic element.
8. The apparatus of claim 1, further comprising a visual tracker to determine the change of pose.
9. The apparatus of claim 1, wherein the haptic element is mechanically attached to a movable robotic element and the movable robotic element is arranged in a field of view of the camera.
10. A method, comprising:
- providing a haptic element comprising at least one passive elastically deformable element deformable in multiple dimensions;
- capturing images of the elastically deformable element with at least one camera;
- determining a change of pose of the elastically deformable element by visual tracking; and
- determining a measurement of force applied to the elastically deformable element based on the captured images and the determined change of pose, wherein the measurement of force includes at least three dimensions including both force and torque components.
11. The method of claim 10, wherein the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions.
12. The method of claim 10, wherein the step of determining the change of pose comprises observing at least two points on the haptic element.
13. The method of claim 10, wherein the step of determining the change of pose comprises observing three or more points on the haptic element.