Mindstorm:n-point image deformation tool
- 1 Introduction
- 2 Control points manipulation
- 3 Mesh manipulation
- 4 Deformation control
- 5 Undo/redo
- 6 Settings
This is a description of planned features of n-point deformation tool. It should lead to UI specification.
At the beginning, an image is opened and user selects n-point deformation tool. When he clicks on the image, mesh is automatically created above it. User can add points onto image and manipulate with them. He can also manipulate with mesh and control the deformation process – more below.
Mesh above the image is created automatically with use of alpha channel. When there isn't any alpha channel, mesh is created above the whole image.
The deformation method uses two meshes. One is called 'source pose' mesh, the other is called 'destination pose' mesh. Source pose mesh is always hidden and destination pose mesh can be visible. Only destination pose mesh is deformed.
Mesh (lattice) for this method usually consists of squares. The deformation method is not limited to square mesh but square mesh can be constructed in an easier way, usually with less elements (than triangle mesh), and with small tweaks we can achieve similar precision as with triangular mesh.
For this method it holds that with decreasing lattice square size (i.e. increasing number of mesh elements), precision of result is increasing, but inefficiency of the algorithm is increasing as well. User should have an option to adjust the balance.
Control points manipulation
Add control points
With a click at some position on image, control point is added. User can add control points at arbitrary positions on image (except position [or some radius around it which depends on mesh square size] where a point is already placed) – mesh will be adjusted according to that.
Selection, deselection, multi-selection
More or less the same as sections with same names in Cage tool specification.
Delete control points
Selected control points are deleted by <delete> key. Last added point can be deleted by <backspace> key.
Deformation of the image
When user moves (click – drag – release) a control point (or multi-selected control points), the image is deformed.
Depth of control point
There should be an option to adjust which part of overlapping mesh/image is visible – e.g. we have an image of a person (as in the picture below) and we want to move her arm so that it is in front of (or behind) her body.
Every control point has its depth which affects a part of mesh under the control point.
User can change the depth using two buttons – one increases the depth, the other decreases it – or using two keys - which?.
When more control points are selected, every one's depth is increased/decreased.
In some situations it's useful to see how the mesh looks like. User can toggle mesh visibility using a check button (or switch).
Mesh element size
User can choose the size of mesh element (square) using a slider – but only before he starts adding control points. It is because of the fact that when changing mesh square size, we have to recreate 'source pose' and 'destination pose' meshes. It's often impossible to recreate 'destination pose' mesh so that it looks similarly to previous 'destination pose' mesh.
What would be a suitable name for this option? Density or precision?
Deformation algorithm control
The deformation algorithm iteratively solves a least squares problem described in the paper. During that process we periodically display current state of the solution and user can see that the deformation is evolving somehow. Usually hundreds of iterations are performed during 1 FPS. We don't know how much time the solving will take and at the same time we often don't want to find the final solution – we're interested just in the process.
Hence there should be an option to temporary stop the process at some state or let it continue with solving further. How to do that in the best way? It could be done for example by <space> key.
When user is satisfied with result, he presses <enter> key. Deformed image is redrawn using selected interpolation method. The deformation ends.
When user wants to cancel the deformation, he presses <escape> key. Deformed image is discarded, original image appears on the canvas. The deformation ends.
The method allows 3 types of deformation:
- rigid (known as as-rigid-as-possible)
- similar (known as as-similar-as-possible)
- stretch (ARAP combined with weights from Moving Least Squares method)
'Rigid' type is very good for bending objects or manipulating articulated objects. It's not so good in stretching or shrinking of objects. 'Similar' type allows changing of scale. Only translation, rotation and change of scale is included in deformation. It can be seen in the example. 'Stretch' type allows better stretching of objects. It can be seen in the example.
User can select the deformation type using drop down list or radio buttons.
Number of deformation iterations
User can select using slider, the number of iterations that are performed during 1 FPS. The larger the number the more rigid the deformation behaves.
How to call this option? It can be called for example rigidity or flexibility.
Type of interpolation
User can select the type of interpolation which is used when the deformation is terminated in favor of deformed image.
Types of interpolation:
- nearest neighbor (none)