Mindstorm:n-point image deformation tool
- 1 User task list
- 2 Introduction
- 3 Control points manipulation
- 4 Mesh manipulation
- 5 Deformation control
- 6 Undo/redo
- 7 Settings
User task list
- control points manipulation
- add control points
- select (multiple) control points
- remove selected control points
- move selected control points, i.e. deform the image
- change depth of selected control points
- mesh manipulation
- toggle mesh visibility (show, hide)
- change mesh element size
- modify (automatically) created mesh – cut an edge
- deformation control
- stop computation or continue with it
- deformation termination
- deformation mode
- rigid, similar, stretch
- number of deformation iterations
- type of interpolation
- nearest neighbor (none), linear etc.
- deformation mode
The deformation method uses two meshes. One is called 'reference pose' mesh, the other is called 'current pose' mesh. Reference pose mesh is always hidden and current pose mesh can be visible. Only current pose mesh is deformed.
Mesh (lattice) for this method usually consists of squares. The deformation method is not limited to square mesh but square mesh can be constructed in an easier way, usually with less elements (than triangle mesh), and with small tweaks we can achieve similar precision as with triangular mesh.
For this method it holds that with decreasing lattice square size (i.e. increasing number of mesh elements), precision of result is increasing, but inefficiency of the algorithm is increasing as well. User should be able to adjust the balance.
At the beginning, user selects n-point deformation tool. When there is a selected region, deformation is performed only on it.
Mesh above the image is created automatically using alpha channel. When there isn't any alpha channel (or selection), mesh is created above the whole image (or selection).
Control points manipulation
Add control points
With a click at some position on image, control point is added. User can add control points at arbitrary positions on image (except position [or some radius around it which depends on mesh square size] where a point is already placed) – mesh will be adjusted according to that.
Selection, deselection, multi-selection
Delete control points
Selected control points are deleted by <delete> key. Last added point can be deleted by <backspace> key.
Deformation of the image
When user moves (click – drag – release) a control point (or multi-selected control points), the image is deformed.
Depth of control point
There should be an option to adjust which part of overlapping mesh/image is visible – e.g. we have an image of a person (as in ...) and we want to move her arm so that it is in front of (or behind) her body.
Every control point has its depth which affects a part of mesh under the control point.
User can change the depth using two buttons – one increases the depth, the other decreases it – or using two keys - which?.
When more control points are selected, every one's depth is increased/decreased.
In some situations it's useful to see how the mesh looks like. User can toggle mesh visibility using a check button (or switch).
Mesh element size
Note that this option changes both 'reference pose' and 'current pose' mesh element size.
User can choose the size of 'reference pose' mesh element using a slider – but only before he starts the deformation process, i.e. moving points (maybe even before he starts adding control points). It is because of the fact that when changing 'reference pose' mesh square size, it's often impossible to recreate 'current pose' mesh so that it creates the same deformation as previous 'current pose' mesh.
What would be a suitable name for this option? Density or precision?
Deformation algorithm control
The deformation algorithm iteratively solves a least squares problem described in the paper. During that process we periodically display current state of the solution and user can see that the deformation is evolving somehow. Usually hundreds of iterations are performed during 1 FPS. We don't know how much time the solving will take and at the same time we often don't want to find the final solution – we're interested just in the process.
Hence there should be an option to temporary stop the process at some state or let it continue with solving further. How to do that in the best way? It could be done for example by <space> key.
When user is satisfied with result, he presses <enter> key. Deformed image is redrawn using selected interpolation method. The deformation ends.
When user wants to cancel the deformation, he presses <escape> key. Deformed image is discarded, original image appears on the canvas. The deformation ends.
The method allows 3 types of deformation:
- rigid (known as as-rigid-as-possible)
- similar (known as as-similar-as-possible)
- stretch (ARAP combined with weights from Moving Least Squares method)
'Rigid' type is very good for bending objects or manipulating articulated objects. It's not so good in stretching or shrinking of objects. 'Similar' type allows changing of scale. Only translation, rotation and change of scale is included in deformation. It can be seen in the example. 'Stretch' type allows better stretching of objects. It can be seen in the example.
User can select the deformation type using drop down list or radio buttons.
Number of deformation iterations
User can select using slider, the number of iterations that are performed during 1 FPS. The larger the number the more rigid the deformation behaves.
How to call this option? It can be called for example rigidity or flexibility.
Type of interpolation
User can select the type of interpolation which is used when the deformation is terminated in favor of deformed image.
Types of interpolation:
- nearest neighbor (none)