Merge branch 'master' of https://github.com/CMU-Graphics/Scotty3D into master

docs/animation/generate_videos.md

+---
+layout: default
+title: "Generating Videos"
+permalink: /generate_videos/
+---
+
+# Generating Videos
+
+## Converting frames to video with ffmpeg
+
+Once you've rendered all the frames of your video, you can combine them into a video by using:
+
+`ffmpeg -r 60 -f image2 -s 800x600 -i ./Video_[TIME]_%4d.png -vcodec libx264 out.mp4`
+
+Where `[TIME]` should be replaced by the timestamp in each frame's filename (they should all be identical). If you don't have ffmpeg installed on your system, you can get it through most package managers, or you can [download it directly](https://ffmpeg.org/download.html), or use your own video creation program.

docs/animation/overview.md

@@ -6,4 +6,14 @@ permalink: /animation/
 
 # Animation Overview
 
-As always, be mindful of the [project philosophy](../philosophy).
+There are four primary components that must be implemented to support Animation functionality.
+
+- [(Task 1) Spline Interpolation](camera_rays.md)
+- [(Task 2) Skeleton Kinematics](intersecting_objects.md)
+- [(Task 3) Linear Blend Skinning]()
+- [(Task 4) Physical Simulation]()
+
+Each of these components is described at the linked page.
+
+Additionally, the assignment specification asks you to create a video of your animation. Some brief technical notes on creating videos from images are [here](generate_videos.md).
+

docs/animation/skeleton_kinematics.md

+---
+layout: default
+title: "Skeleton Kinematics"
+permalink: /animation/skeleton_kinematics
+---
+
+# Skeleton Kinematics
+
+A `Skeleton`(defined in `scene/skeleton.h`) is what we use to drive our animation. You can think of them like the set of bones we have in our own bodies and joints that connect these bones. For convenience, we have merged the bones and joints into the `Joint` class which holds the orientation of the joint relative to its parent as euler angle in its `pose`, and `extent` representing the direction and length of the bone with respect to its parent `Joint`. Each `Mesh` has an associated `Skeleton` class which holds a rooted tree of `Joint`s, where each `Joint` can have an arbitrary number of children.
+
+All of our joints are ball `Joint`s which have a set of 3 rotations around the [[task2_media/0027.png|height=9px]], [[task2_media/0028.png|height=12px]], and [[task2_media/0029.png|height=9px]] axes, called _Euler angles_. Whenever you deal with angles in this way, a fixed order of operations must be enforced, otherwise the same set of angles will not represent the same rotation. In order to get the full rotational transformation matrix, [[task2_media/0030.png|height=14px]], we can create individual rotation matrices around the [[task2_media/0031.png|height=14px]], [[task2_media/0032.png|height=14px]], and [[task2_media/0033.png|height=14px]] axes, which we call [[task2_media/0034.png|height=16px]], [[task2_media/0035.png|height=18px]], and [[task2_media/0036.png|height=16px]] respectively. The particular order of operations that we adopted for this assignment is that [[task2_media/0037.png|height=18px]].
+
+### Forward Kinematics
+
+_Note: These diagrams are in 2D for visual clarity, but we will work with a 3D kinematic skeleton._
+
+When a joint's parent is rotated, that transformation should be propagated down to all of its children. In the diagram below, [[task2_media/0038.png|height=12px]] is the parent of [[task2_media/0039.png|height=12px]] and [[task2_media/0040.png|height=12px]] is the parent of [[task2_media/0041.png|height=12px]]. When a translation of [[task2_media/0042.png|height=12px]] and rotation of [[task2_media/0043.png|height=16px]] is applied to [[task2_media/0044.png|height=12px]], all of the descendants are affected by this transformation as well. Then, [[task2_media/0045.png|height=12px]] is rotated by [[task2_media/0046.png|height=16px]] which affects itself and [[task2_media/0047.png|height=12px]]. Finally, when rotation of [[task2_media/0048.png|height=16px]] is applied to [[task2_media/0049.png|height=12px]], it only affects itself because it has no children.
+
+[[media/forward_kinematic_diagram.jpg]]
+
+You need to implement these routines in `student/skeleton.cpp` for forward kinematics.
+
+*   `Joint::joint_to_bind` 
+    Rreturn a matrix transforming points in the space of this joint
+    to points in mesh space in bind position  up to the base of this joint (end of its parent joint). You should traverse upwards from this joint's parent all the way up to the root joint and accumulate their transformations. 
+*   `Joint::joint_to_posed`
+    Return a matrix transforming points in the space of this joint to points in mesh space, taking into account joint poses. Again, you should traverse upwards from this joint's parent to the root joint.
+*   `Skeleton::end_of`
+    Returns the end position of the joint in world coordinate frame, and you should take into account the base position of the skeleton (`Skeleton::base_pos`).
+*   `Skeleton::posed_end_of`
+    Returns the end position of the joint in world coordinate frame with poses, and you should take into account `Skeleton::base_pos`.
+*   `Skeleton::joint_to_bind` 
+    Rreturn a matrix transforming points in the space of this joint
+    to points in mesh space in bind position but with the base position of the skeleton taken in to account. Hint: use some function that you have implemented wisely!
+*   `Skeleton::joint_to_posed`
+    Return a matrix transforming points in the space of this joint to points in mesh space, taking into account joint poses but with the base position of the skeleton taken in to account. Hint: use some function that you have implemented wisely!
+
+Once you have implemented these basic kinematics, you should be able to define skeletons, set their positions at a collection of keyframes, and watch the skeleton smoothly interpolate the motion (see the [user guide](../guide/animate.md) for an explanation of the interface). The gif below shows a very hasty demo defining a few joints and interpolating their motion.
+
+![gif1](task2_media/gif1.gif)
+
+![gif1](task2_media/gif2.gif)
+
+Note that the skeleton does not yet influence the geometry of the cube in this scene -- that will come in Task 3!
+
+<!---
+### Task 2b - Inverse Kinematics
+
+#### Single Target IK
+
+Now that we have a logical way to move joints around, we can implement Inverse Kinematics, which will move the joints around in order to reach a target point. There are a few different ways we can do this, but for this assignment we'll implement an iterative method called gradient descent in order to find the minimum of a function. For a function [[task2_media/0050.png|height=16px]], we'll have the update scheme:
+
+[[task2_media/0051.png|height=18px]]
+
+Where [[task2_media/0052.png|height=9px]] is a small timestep. For this task, we'll be using gradient descent to find the minimum of the cost function:
+
+[[task2_media/0053.png|height=36px]]
+
+Where [[task2_media/0054.png|height=19px]] is the position in world space of the target joint, and [[task2_media/0055.png|height=12px]] is the position in world space of the target point.
+
+More specifically, we'll be using a technique called Jacobian Transpose, which relies on the assumption that:
+
+[[task2_media/0056.png|height=21px]]
+
+Where:
+
+*   [[task2_media/0057.png|height=14px]] (n x 1) is the function [[task2_media/0058.png|height=19px]], where [[task2_media/0059.png|height=19px]] is the angle of joint [[task2_media/0060.png|height=14px]] around the axis of rotation
+*   [[task2_media/0061.png|height=9px]] is a constant
+*   [[task2_media/0062.png|height=16px]] (3 x n) is the Jacobian of [[task2_media/0063.png|height=14px]]
+
+Note that here [[task2_media/0064.png|height=9px]] refers to the number of joints in the skeleton. Although in reality this can be reduced to just the number of joints between the target joint and the root, inclusive, because all joints not on that path should stay where they are, so their columns in [[task2_media/0065.png|height=16px]] will be 0\. So [[task2_media/0066.png|height=9px]] can just be the number of joints between the target and the root, inclusive. Additionally note that since this will get multiplied by [[task2_media/0067.png|height=9px]] anyways, you can ignore the value of [[task2_media/0068.png|height=9px]], and just consider the timestep as [[task2_media/0069.png|height=18px]].
+
+Now we just need a way to calcluate the Jacobian of [[task2_media/0070.png|height=14px]]. For this, we can use the fact that:
+
+[[task2_media/0071.png|height=19px]]
+
+Where:
+
+*   [[task2_media/0072.png|height=16px]] is the [[task2_media/0073.png|height=16px]] column of [[task2_media/0074.png|height=19px]]
+*   [[task2_media/0075.png|height=14px]] is the axis of rotation
+*   [[task2_media/0076.png|height=16px]] is the vector from the base of joint [[task2_media/0077.png|height=14px]] to the end point of the target joint
+
+For a more in-depth derivation of Jacobian transpose (and a look into other inverse kinematics algorithms), please check out [this presentation](https://web.archive.org/web/20190501035728/https://autorob.org/lectures/autorob_11_ik_jacobian.pdf). (Pages 45-56 in particular)
+
+Now, all of this will work for updating the angle along a single axis, but we have 3 axes to deal with. Luckily, extending it to 3 dimensions isn't very difficult, we just need to update the angle along each axis independently.
+
+#### Multi-Target
+
+We'll extend this so we can have multiple targets, which will then use the function to minimize:
+
+[[task2_media/0078.png|height=40px]]
+
+which is a simple extension actually. Since each term is independent and added together, we can get the gradient of this new cost function just by summing the gradients of each of the constituent cost functions!
+
+You should implement multi-target IK, which will take a `std::map` of `Joint`s and target points for that joint. Each joint can only have 1 target point.
+
+In order to implement this, you should update `Joint::calculateAngleGradient` and `Skeleton::reachForTarget`. `Joint::calculateAngleGradient` should calculate the gradient of [[task2_media/0079.png|height=14px]] in the x,y, and z directions, and add them to `Joint::ikAngleGradient` for all relevant joints. `Skeleton::reachForTarget` should actually do the gradient descent calculations and update the angles of each joint, saving them with `Joint::setAngle`. In this function, you should probably use a very small timestep, but do several iterations (say, 10s to 100s) of gradient descent in order to speed things up. For even faster and better results, you can also implement a variable timestep instead of just using a fixed one. Note also that the root joint should never be updated.
+
+A key thing for this part is to _remember what coordinate frame you're in_, because if you calculate the gradients in the wrong coordinate frame or use the axis of rotation in the wrong coordinate frame your answers will come out very wrong!
+
+#### Using your IK!
+Once you have IK implemented, you should be able to create a series of joints, and get a particular joint to move to the desired final position you have selected.
+
+[[media/ik.gif]]
+-->
\ No newline at end of file

docs/animation/skinning.md

+---
+layout: default
+title: "Animation Overview"
+permalink: /animation/skinning
+---
+
+# Linear Blend Skinning
+
+Now that we have a skeleton set up, we need to link the skeleton to the mesh in order to get the mesh to follow the movements of the skeleton. We will implement linear blend skinning using the following functions: `Skeleton::skin()`, `Skeleton::find_joints()`, and `closest_on_line_segment`.
+
+The easiest way to do this is to update each of mesh vertices' positions in relation to the bones (Joints) in the skeleton. There are 3 types of coordinate spaces: bind, joint, and pose. Bind is the initial coordinate frame of the vertices of where they are bound to relative to the mesh. Joint is the position of the vertex relative to a given joint. Pose is the world-spacce position after the joint transforms have been applied. You'll want to compute transforms that take vertices in bind space and convert them to posed space (Hint: `joint_to_bind`, `joint_to_posed`, and `inverse()` will come in handy.)
+
+Your implementation should have the following basic steps for each vertex:
+
+
+- Compute the vertex's position with respect to each joint j in the skeleton in j's coordinate frame when no transformations have been applied to the skeleton (bind pose, vertex bind position).
+- Find where this vertex would end up (in world coordinates) if it were transformed along with bone j.
+- Find the closest point on joint j's bone segment (axis) and compute the distance to this closest point (Hint: `closest_on_line_segment` might come in handy).
+- Compute the resulting position of the vertex by doing a weighted average of the bind-to-posed transforms from each bone and applying it to the vertex. The weights for the weighted average should be the inverse distance to the joint, so closer bones have a stronger influence.
+
+Below we have an equation representation. The ith vertex v is the new vertex position. The weight w is the weight metric computed as the inverse of distance between the ith vertex and the closest point on joint j. We multiply this term with the position of the ith vertex v with respect to joint j after joint's transformations has been applied.
+
+![skinnning_equations](task3_media/skinning_equations.png)
+
+In Scotty3D, the `Skeleton::skin()` function gets called on every frame draw iteration, recomputing all skinning related quantities. In this function, you should read vertices from `input.verts()` and indices from `input.indices()`, and write the resulting positions and norms to `v.pos` and `v.norm` for every vertex in the input vertices list. 
+
+You will be implementing a Capsul-Radius Linear Blend Skin method, which only moves vertices with a joint if they lie in the joint's radius. The `Skeleton::skin()` function also takes in a `map` of vertex index to relevant joints that you must compute the above distance/transformation metrics on. You are also responsible for creating this `map`, which is done so in `Skeleton::find_joints()`. Don't worry about calling this function, it is called automatically before skin is called, populating the `map` field and sending it over to the `skin()` function. Your `Skeleton::find_joints()` implementation should iterate over all the vertices and add joint j to vertex index i in the map if the distance between the vertex and joint is less than `j->radius` (remember make sure they're both in the same coordinate frame.)
+
+![skinning](task3_media/skinning.gif)
\ No newline at end of file

docs/animation/splines.md

+---
+layout: default
+title: "Animation Overview"
+permalink: /animation/splines
+---
+
+As we discussed in class, data points in time can be interpolated by constructing an approximating piecewise polynomial or spline. In this assignment you will implement a particular kind of spline, called a Catmull-Rom spline. A Catmull-Rom spline is a piecewise cubic spline defined purely in terms of the points it interpolates. It is a popular choice in real animation systems, because the animator does not need to define additional data like tangents, etc. (However, your code may still need to numerically evaluate these tangents after the fact; more on this point later.) All of the methods relevant to spline interpolation can be found in `spline.h` with implementations in `spline.cpp`.
+
+### Task 1a - Hermite Curve over the Unit Interval
+
+Recall that a cubic polynomial is a function of the form
+
+<img src=task1_media/0000.png height="30">
+
+where <img src=task1_media/0001.png height="20">, and <img src=task1_media/0002.png height="20"> are fixed coefficients. However, there are many different ways of specifying a cubic polynomial. In particular, rather than specifying the coefficients directly, we can specify the endpoints and tangents we wish to interpolate. This construction is called the "Hermite form" of the polynomial. In particular, the Hermite form is given by
+
+<img src=task1_media/0003.png height="30">
+
+where <img src=task1_media/0004.png height="20"> are the endpoint positions, <img src=task1_media/0005.png height="20"> are the endpoint tangents, and <img src=task1_media/0006.png height="20"> are the Hermite bases
+
+<img src=task1_media/0007.png height="30"> <br/>
+
+<img src=task1_media/0008.png height="30"> <br/>
+
+<img src=task1_media/0009.png height="30"> <br/>
+
+<img src=task1_media/0010.png height="30"> <br/>
+
+Your first task is to implement the method `Spline::cubic_unit_spline()`, which evaluates a spline defined over the time interval <img src=task1_media/0011.png height="20"> given a pair of endpoints and tangents at endpoints. Optionally, the user can also specify that they want one of the time derivatives of the spline (1st or 2nd derivative), which will be needed for our dynamics calculations.
+
+Your basic strategy for implementing this routine should be:
+
+*   Evaluate the time, its square, and its cube (for readability, you may want to make a local copy).
+*   Using these values, as well as the position and tangent values, compute the four basis functions <img src=task1_media/0012.png height="20"><img src=task1_media/0013.png height="20"> of a cubic polynomial in Hermite form. Or, if the user has requested the nth derivative, evaluate the nth derivative of each of the bases.
+*   Finally, combine the endpoint and tangent data using the evaluated bases, and return the result.
+
+Notice that this function is templated on a type T. In C++, a templated class can operate on data of a variable type. In the case of a spline, for instance, we want to be able to interpolate all sorts of data: angles, vectors, colors, etc. So it wouldn't make sense to rewrite our spline class once for each of these types; instead, we use templates. In terms of implementation, your code will look no different than if you were operating on a basic type (e.g., doubles). However, the compiler will complain if you try to interpolate a type for which interpolation doesn't make sense! For instance, if you tried to interpolate `Skeleton` objects, the compiler would likely complain that there is no definition for the sum of two skeletons (via a + operator). In general, our spline interpolation will only make sense for data that comes from a vector space, since we need to add T values and take scalar multiples.
+
+### Task 1B: Evaluation of a Catmull-Rom spline
+
+The routine from part 1A just defines the interpolated spline between two points, but in general we will want smooth splines between a long sequence of points. You will now use your solution from part 1A to implement the method `Spline::at()` which evaluates a general Catmull-Romspline (and possibly one of its derivatives) at the specified time in a sequence of points (called "knots"). Since we now know how to interpolate a pair of endpoints and tangents, the only task remaining is to find the interval closest to the query time, and evaluate its endpoints and tangents.
+
+The basic idea behind Catmull-Rom is that for a given time t, we first find the four closest knots at times
+
+<img src=task1_media/0014.png height="30">
+
+We then use t1 and t2 as the endpoints of our cubic "piece," and for tangents we use the values
+ome to Scotty3D! This 3D graphics software implements interactive mesh editing, realistic path tracing, and dynamic animation. Implementing the functionality of the program constitutes the majority of the coursework for 15-462/662 Computer Graphics at Carnegie Mellon University
+<img src=task1_media/0015.png height="30"> <br/>
+
+<img src=task1_media/0016.png height="30"> <br/>
+
+In other words, a reasonable guess for the tangent is given by the difference between neighboring points. (See the Wikipedia and our course slides for more details.)
+
+<img src=task1_media/spline_diagram.jpg height="30"> <br/>
+
+This scheme works great if we have two well-defined knots on either side of the query time t. But what happens if we get a query time near the beginning or end of the spline? Or what if the spline contains fewer than four knots? We still have to somehow come up with a reasonable definition for the positions and tangents of the curve at these times. For this assignment, your Catmull-Rom spline interpolation should satisfy the following properties:
+
+*   If there are no knots at all in the spline, interpolation should return the default value for the interpolated type. This value can be computed by simply calling the constructor for the type: T(). For instance, if the spline is interpolating Vector3D objects, then the default value will be <img src=task1_media/0017.png height="20">.
+*   If there is only one knot in the spline, interpolation should always return the value of that knot (independent of the time). In other words, we simply have a constant interpolant. (What, therefore, should we return for the 1st and 2nd derivatives?)
+*   If the query time is less than or equal to the initial knot, return the initial knot's value. (What do derivatives look like in this region?)
+*   If the query time is greater than or equal to the final knot, return the final knot's value. (What do derivatives look like in this region?)
+
+Once we have two or more knots, interpolation can be handled using general-purpose code. In particular, we can adopt the following "mirroring" strategy to obtain the four knots used in our computation:
+
+*   Any query time between the first and last knot will have at least one knot "to the left" <img src=task1_media/0018.png height="20"> and one "to the right" <img src=task1_media/0019.png height="20">.
+*   Suppose we don't have a knot "two to the left" <img src=task1_media/0020.png height="20">. Then we will define a "virtual" knot <img src=task1_media/0021.png height="20">. In other words, we will "mirror" the difference be observe between <img src=task1_media/0022.png height="20"> and <img src=task1_media/0023.png height="20"> to the other side of <img src=task1_media/0024.png height="20">.
+*   Likewise, if we don't have a knot "two to the right" <img src=task1_media/0025.png height="20">), then we will "mirror" the difference to get a "virtual" knot <img src=task1_media/0026.png height="20">.
+*   At this point, we have four valid knot values (whether "real" or "virtual"), and can compute our tangents and positions as usual.
+*   These values are then handed off to our subroutine that computes cubic interpolation over the unit interval.
+
+An important thing to keep in mind is that `Spline::cubic_unit_spline()` assumes that the time value t is between 0 and 1, whereas the distance between two knots on our Catmull-Rom spline can be arbitrary. Therefore, when calling this subroutine you will have to normalize t such that it is between 0 and 1, i.e., you will have to divide by the length of the current interval over which you are interpolating. You should think very carefully about how this normalization affects the value and derivatives computed by the subroutine, in comparison to the values and derivatives we want to return. A transformation is necessary for both the tangents that you feed in to specify the unit spline, as well as derivatives that are returned from the unit spline (but not necessarily the same transformation!).
+
+Internally, a Spline object stores its data in an STL map that maps knot times to knot values. A nice thing about an STL map is that it automatically keeps knots in sorted order. Therefore, we can quickly access the knot closest to a given time using the method `map::upper_bound()`, which returns an iterator to knot with the smallest time greater than the given query time (you can find out more about this method via online documentation for the Standard Template Library).
+
+### Using the splines
+
+Once you have implemented the functions in `spline.cpp`, you should be able to make simple animations by translating, rotating or scaling the mesh in the scene. The main idea is to:
+* create an initial keyframe by clicking at a point on the white timeline at the bottom of the screen
+* specify the initial location/orientation/scale of your mesh using the controls provided
+* create more keyframes with different mesh locations/orientations/scales and watch the splines smoothly interpolate the movement of your mesh!
+
+
+
+<img src=task1_media/animate_cow.gif>
\ No newline at end of file