Here is a quick video of the results before we get on to the discussion:
Basically this started because a colleague at Georgia Tech, Alex Trevor (soon to be Dr,), had an Oculus Rift and mounted a camera on top of it to see the Kinect output with the Oculus Rift. I thought that was really cool and had previously worked on a game that used the Kinect Skeleton to throw fireballs (though I lost all the source code for the first version). I really wanted to combine those ideas and had to start over.
Luckily, Dr. Brian Peasley (now at Microsoft) came to my rescue as always and gifted me an Oculus Rift.
Luckily, Dr. Brian Peasley (now at Microsoft) came to my rescue as always and gifted me an Oculus Rift.
To explain a bit, the Oculus Rift is an immersive virtual reality headset. The two images are displayed on the screen above because one is projected to the left eye and one to the right eye. As you rotate your head, the images change and it feels like you are looking at a real environment (it is pretty amazing). This is why there are two images in the video (and pretty much all Oculus Rift demos). To appreciate it fully, I recommend wearing an Oculus Rift while watching the video.
The Kinect everyone should know by now. It can yield a really good skeletal estimation of a persons joints from the depth data, which can then be used to represent gestures.
I had some time this week so I grabbed the Unity third person MMO example, added the Kinect scripts provided by CMU here, created my own fireball and fireball related prefabs and scripts, added the Oculus package, changed all the camera stuff to make it first person, and tweaked a lot of stuff. Daniel Castro (also at Georgia Tech) was nice enough to help me film me making a fool of myself.
It was obviously a bit more complicated than that. Lots of tinkering and scripting was involved to get things working but that is the output. Here at RIM, we are working on lots of other cool things and if you are interested, check out some of my other projects.
Ive uploaded the source for public use and you can find it here (just make sure to please cite me):
https://github.com/StevenHickson/UnityKinectOculus
The fireball script just takes two Game objects (the Vikings left hand normalized by your hip) and uses a velocity measurement to determine if you want to throw the fireball, then it creates a fireball and sets its velocity to your hands velocity whenever you throw. This can be done easily with a small amount of code as below:
void Update () {
Vector3 norm_hand = HandPosition.transform.position - HipPosition.transform.position;
Vector3 velocity = (norm_hand - lastPos) / Time.deltaTime;
float dist = velocity.magnitude;
if (Input.GetButtonDown("Fire1") || (dist > THRESH && dist < MAX_THRESH)) {
Rigidbody clone;
Vector3 pos = HandPosition.transform.position;
pos.z += 1;
clone = (Rigidbody)Instantiate(Projectile, pos, transform.rotation);
clone.velocity = velocity * SPEED;
}
lastPos = norm_hand;
}
And thats it for the fireball. Then there are some scripts destroying the fireball and vikings when they collide.
For the mapping of the joints to the main Viking, each joint position of the Viking is mapped to the corresponding Kinect Skeleton joints with GameObjects in the KinectControllerScript.
Then the Oculus SDK is used to create the camera and player control mapped to the main viking.
Then the Oculus SDK is used to create the camera and player control mapped to the main viking.
For all the code, see the Github project
Im using a friends version of Unity Pro because Im a poor graduate student. So please donate if you liked this work so I can continue doing it. All of these gadgets are expensive and I do all this and post it for free.
