Curve Tracking and Analysis in
3-D
While the bulk of SynthEyes is concerned with determining the location of
points in 3-D, sometimes it can be essential to determine the
shape of a curve in 3-D, even if that curve has no trackable points on it, and
every point along the curve appears the same as every other. For example, it
might be the curve of a highway overpass to which a car chase must be added,
the shape of a window opening on a car, or the shape of a sidewalk on a hilly
road, which must be used as a 3-D masking edge for an architectural
insert.
In such situations, acquiring the 3-D shape can be a tremendous advantage,
and SynthEyes can now bring it to you using its novel curve tracking and flex
solving capability, as operated with the
Flex/Curve Control Panel
.Terminology
There's a bit of new terminology to define here, since there are both 2-D
and 3-D curves being considered.
Curve. This refers to a spline-like 2-D curve. It will
always live on one particular shot's images, and is animated with a different
location on each frame.
Flex. A spline-like 3-D curve. A flex
resides in 3-D, though it may be attached to a moving object. One or more
curves will be attached to the flex; those curves will be analyzed to determine
the 3-D shape of the flex.
Rough-in. Placing control-point keys periodically and
approximately.
Tuning a curve. Adjusting a curve so it matches edges
exactly.
Overview
Here's the overall process for using the curve and flex system to
determine a 3-D curve. The quick synopsis is that we will get the 2-D curves
positioned exactly on each frame throughout the shot, then run a 3-D solving
stage. Note that the ordering of the steps can be changed around a bit, and
additional wrinkles added, once you know what you are doing — this is the
simplest and easiest to explain.
1. Open the shot in SynthEyes
2. Obtain a 3-D camera solution, using
automatic or supervised tracking
3. At the beginning of the shot, create a
(2-D) curve corresponding to the flex-to-be.
4. “Rough-in” the path of the curve, with
control-point animation keys throughout the shot. There is a tool that can help
do this, using the existing point trackers.
5. Tune the curve to precisely match the
underlying edges (manual or automatic).
6. Draw a new flex in an approximate
location. Assign the curve to it.
7. Configure the handling of the ends of the
flex.
8. Solve the flex
9. Export the flex or convert it to a series
of trackers.
Shot Planning and Limitations
Determining the 3-D position of a curve is at the mercy of underlying
mathematics, just as is the 3-D camera analysis performed by the rest of
SynthEyes. Because every point along a curve/flex is equivalent, there is
necessarily less information in the curve data than in a collection of
trackers.
As a result, first, flex analysis can only be performed
after a successful normal 3-D solve that has determined camera
path and field of view. The curve data can not help obtain that solve; it does
not replace and is not equivalent to the data of several trackers.
Additionally, the camera motion must be richer and more complex than for a
collection of trackers. Consider a flex consisting of a horizontal line,
perhaps a clothesline or the top of a fence. If the camera moves left to right
so that its path is parallel to the flex, no 3-D information (depth) can be
produced for the flex. If the camera moves vertically, then the depth
information can be obtained. The situation is reversed for a vertical line: a
vertical camera motion will not produce any depth information.
Generally, both the shape of the flex and camera path will be more
complex, and you will need to ensure that the camera path is sufficiently
complex to produce adequate depth information for all of the flex. If the flex
is circular, and the camera motion horizontal, then the top and bottom of the
circle will not have well-defined depth. The flex will prefer a flat
configuration, which is often, but not necessarily, correct.
Note that a simple diagonal motion will not solve this problem: it will
not explore the depth in the portion of the circle that is parallel to the
motion path. The camera path must itself curve to more completely identify the
depth all the way around the circle — hence the comment that the camera motion
must itself be more complex than for point tracking.
Similarly, tripod (nodal pan) shots are not suitable for use with the
curve & flex solving system. As with point tracking, tripod shots do not
produce any depth information.
Flexes and curves are not closed like the letter O — they are open like
the letter U or C. Also, they do not contain corners, like a V. Nor do they
contain tangency handles, since the curvature is controlled by SynthEyes.
Generally, the curve will be set up to track a fairly visible edge in the
image. Very marginal edges can still be used and solved to produce a flex, if
you are willing to do the tracking by hand.
Initial Curve Setup
Once you have identified the section of curve to be tracked and made into
a 3-D flex, you should open the Flex Control Panel
,
which contains both flex and curve controls, and select the camera
view.Click the New Curve button, then, in the Camera View, click along the
section of curve to be tracked, creating control points as you go. Place
additional control points in areas of rapid curvature, and at extremal points
of the curve. Avoid area where there is no trackable edge if possible.
When you have finished with the last control point, right-click to exit
the curve creation mode.
Roughing in the Curve Keys
Next, we will approximately position the curve to track the underlying
edge. This can be done manually or automatically, if the situation
permits.
Manual Roughing
For manual roughing, you move through the shot and periodically re-set the
position of the curve. By starting at the ends, and then successively
correcting the position at the most extremely-wrong positions within the shot,
usually this isn’t too time consuming (unless the shot is a jumpy hand-held
one). SynthEyes splines the control point positions over time.
To re-set the curve, you can drag the entire curve into an approximate
position, then adjust the control points as necessary. If you find you need
additional control points, you can shift-click within the curve to create
them.
You should monitor the control point density so that you don’t bunch many
of them in the same place. But you do not have to worry about control points
“chattering” in position along the curve. This will not affect
SynthEyes or the resulting flex.
Automatic Roughing
SynthEyes can automatically rough the curve into place with a special tool
— as long as there is a collection of trackers around the curve (not just one
end), such that the trackers and curve are all roughly on the same plane.
When this is the case, shift-select all the trackers you want to use,
click the Rough button on the Flex control panel, then click
the curve to be roughed into place.
The Rough Curve Import panel will appear, a simple affair.
The first field asks how many trackers must be valid for the roughing
process to continue. In this case, 5 trackers were selected to start. As shown,
it will continue even if only one is valid. If the value is raised to 5, the
process will stop once any tracker becomes invalid. If only a few trackers are
valid (especially less than 4), less useful predictions of the curve shape can
be made.
The Key every N frames setting controls how often the curve is keyed. At
the default setting of 1, a key will be placed at every frame, which is
suitable for a hand-held shot, but less convenient to subsequently refine. For
a smooth shot, a value of 10-20 might be more appropriate.
The Rough Curve Importer will start at the current frame, and begin
creating keys every so often as specified. It will stop if it reaches the end
of the shot, if there are too few trackers still valid, or if it passes by any
existing key on the curve. You can take advantage of this last point to “fill
in” keys selectively as needed, using different sets of trackers at different
times, for example.
After you’ve used the Rough Curve Import tool, you should scrub through
the shot to look for any places where additional manual tweaking is
required.
The curve may go offscreen or be obscured. If this happens, you can use
the curve Enable checkbox to disable the curve. Note that it is OK if the curve
goes partly offscreen, as long as there is enough information to locate it
while it is onscreen.
Curve Tuning
Once the curve has been roughed into place, you’re ready to “tune” it to
place it more accurately along the edge. Of course, you can do this all by
hand, and in adverse conditions, that may be necessary. But it is much better
to use the automated Tune tool.
You can tune either a single frame, with the Tune button, or all of the
frames using of course the All button. When a curve is tuned
on a frame, the curve control points will latch onto the nearby edge.
For this reason, before you begin tuning, you may wish to create
additional control points along the curve, by shift-clicking it.
The All button will bring up a control panel that controls both the
single- and multi-frame tuning. If you want to adjust the parameters without
tuning all the frames, simply close the dialog instead of hitting its Go
button.
You can adjust to edges of different widths, control the distance within
which the edge is searched, and alter the trade-off between a large distant
edge, and a smaller nearby one. Clearly, it is going to be easier to track
edges with no nearby edges of similar magnitude.
The control panel allows you to tune all frames (potentially just those
within the animation playback range), only the frames that already have keys
(to tune your roughed-in frames), or only the frames that do not have keys (to
preserve your previously-keyed frames).
You can also tell the tracking dialog to use the tuned locations as it
estimates (using splining) where the curve is in subsequent frames, by turning
on the Continuous Update checkbox. If you have a simple curve
well-separated from confounding factors, you can use this feature to track a
curve through a shot without roughing it in first. The drawback of doing this
is that if the curve does get off course, you can wind up with many bad keys
that must be repaired or replaced. [You can remove erroneous keys using
Truncate.] With the Continuous Update box off, the tuning process is more
predictable, relying solely on your roughed-in animation.
Flex Creation
With your curve(s) complete, you can now create a flex, which is the 3-D
splined curve that will be made to match the curve animation. The flex will be
created in 3-D in a position that approximately matches its actual position and
shape. It is usually most convenient to open the Quad view, so that you can see
the camera view at the same time you create the flex in one of the 3-D views
(such as the Top view).
Click the New Flex button, then begin clicking in the chosen 3-D view to
lay out a succession of control points. Right-click to end the mode. You can
now adjust the flex control points as needed to better match the curve. You
should keep the flex somewhat shorter than the curve.
To attach the curve to the flex, select the curve in the camera view,
then, on the flex control panel, change the parent-flex list box for the curve
to be your flex. (Note: if you create a flex, then a curve while the flex is
still selected, the curve is automatically connected to the flex.)
Flex Endpoints
The flex's endpoints must be “nailed down” so that the flex can not just
shrivel up along the length of the curve, or pour off the end. The ends are
controlled by one of several different means:
1. the end of the flex can stay even with its
initial position,
2. the end of the flex can stay even with a
specific tracker, or
3. the end of the flex can exactly match the
position of a tracker.
The first method is the default. The last method is possible only if there
is a tracker at the desired location; this arises most often when several lines
intersect. You can track the intersection, then force all of the flexes to meet
at the same 3-D location.
To set the starting or ending tracker location for a flex, click the Start
Pt or End Pt button, then click on the desired tracker. Note that the current
3-D location of the tracker will be saved, so if you re-track or re-solve, you
will need to reset the endpoint.
The flex will end “even” with the specified point, meaning so that the
point is perpendicular to the end of the flex. To match the position exactly,
turn on the Exact button.
Flex Solving
Now that you’ve got the curve and flex set up, you are ready to solve.
This is very easy — click the Solve button (or Solve All if you have several
flexes ready to be solved).
After you solve a flex, the control points will no longer be visible—they
are replaced by a more densely sampled sequence of non-editable points. If you
want to get back to the original control points to adjust the initial
configuration, you can click Clear.
Flex Exports
Once you have solved the flex, you can export it. At present, there are
two principal export paths. The flexes are not currently exported as part of
regular tracker exports.
First, you can convert the flex into a sequence of trackers with the
Convert Flex to Trackers script on the Script menu. The trackers can be
exported directly, or, more usefully, you can use them in the Perspective
window to create a mesh containing those trackers. For example, on a building
project where the flex is the edge of the road, you can create a ground mesh to
be landscaped, and still have it connect smoothly with the road, even if the
road is not planar.
Second, you can export the coordinates of the points along the flex into a
text file using the Flex Vertex Coordinates exporter. Using that file is up to
you, though it should be possible to use it to create paths in most
packages.
