Re: A question about Virtual Reality.

Hi Jason

Yes, it is possible to record both still images and video in 360-degrees - although exactly what 360-degrees means should really be explained, as it could be misleading. It should also be possible to link a virtual reality headset to control which portion of the recorded information is displayed.

One way of explaining 360-degrees would be to say that it represents all the directions on the compass. Another way is to explain that if you stand on a spot, and then rotate your body on the spot so that you see all around the room, you have seen all 360-degrees of the room. But this doesn't really capture all of the information in the room - we haven't looked up or down. In reality, the are another 360-degrees that we could view - the "looking up and down" directions. A better, more mathematical way of describing these would be to say that there are two sets of directions that one can look in when inside an environment: those around the vertical axis and those around the horizontal axis (and obviously combinations of both).

So how would someone take a "360-degree" photograph or video? If we consider that 360-degrees just means those directions around the vertical axis, then we can take a 360-degree photograph using just an ordinary camera: Just stand up, hold the camera and take a picture, then rotate a few degree so that the camera is pointing in a new direction (but so that this new picture will have a slight overlap with the last picture) and take a picture. Once you have developed the photos from all the directions, you can overlap them to form a long 360-degree image - try it, it's quite fun.

But this doesn't really address your question. What would be nice is to somehow automatically capture all 360-degrees at once, so that you could take a 360-degree photograph/video/film. One of the solutions to the problem is described in your question - use multiple cameras pointing in the required directions (overlapping the cameras' field of view so that the resulting images could be tied together more easily). But this is expensive because you need several cameras, and they would take up a fair bit of space, so the 360-degree camera would be rather heavy and cumbersome.

Another solution would be to use a special lens that captures a 360-degree image. There are a number of companies that produce such lenses, including Be Here (http://www.behere.com/) - pictures of their lenses can be found at http://www.behere.com/prdphoto.ht m.

Some of the lenses that are available not only take 360-degree pictures around the vertical axis (obviously depending on where the lens is pointed!), but also some of the directions around the horizontal axis. Note, however, that if you were to take a picture using one of these lenses, the actual photo once developed would probably look very distorted - a bit like a "fish eye" effect. So in order to be able to view a portion of the image, you would need to convert the distorted image to one that is uniform and makes sense when we look at it. This could possibly be done using another lens, but it is probably easier to digitise the 360-degree image (convert it to a format that a computer can store) and then use a computer to do some maths on the image information to create the uniform image.

There are some quite nice demonstrations of images from these sorts of lenses on the internet. The following site has a 360-degree photograph that you can rotate using the mouse: http://vitalsoft. org.org.mx/~jlab reu/video360.html. A company called iPIX (who provide 360-degree imaging solutions) have a similar demonstration that you can rotate and zoom - but you will need to download an ActiveX Control to view this (it should load automatically just by viewing the page - choose "Yes" to load and install the ActiveX Control when the dialog pops up). The demonstration can be found at: http://www.ipix.com/cho ose_ipix/full360.html.

Be Here (who produce a 360-degree lens) have links to demonstration videos in which you can choose where to look; these links can be found at Be Here Gallery. One of these demonstrations is from the TV show Ally McBeal and is a 360- degree tour of the sets used in the filming of the show. The page can be found at: http://www.allymcbeal.com /toybox/index.htm. You will need RealPlayer to view this, and you may also need to download a plug-in for RealPlayer to allow you to choose the direction to look in (you should be prompted to load the plug-in when clicking on a room in the TV show to view). You can look around the video by clicking and dragging the mouse inside the image. Note that videos take a lot of memory, so if you have a slow internet connection it may take a while for you to be able to view the video.

So now it is possible to take 360-degree pictures, how can we link these pictures to a virtual reality headset so that we can look around the environment we have filmed using the 360-degree lens? First, I'll explain what virtual reality (VR) is, and then I'll suggest how you could link a 360-degree video to a VR headset.

Virtual Reality (see http://we bopedia.internet.com/TERM/v/virtual_reality.html) is the name given to a system that attempts to immerse the user in an environment that mimics real life. The user wears devices that allow them to interact with and experience the virtual environment. A typical VR system will be comprised of a computer that can generate a 3D environment (images, sound etc.), a headset (to view the environment and hear stereo sound), possibly one or more navigation devices (like joysticks or a 3D "mouse"), possibly gloves (so that virtual hands can be used within the environment - see a nimation of VR hands The computer monitors the position of the user within the virtual environment (i.e. the position of the gloves or navigational devices) and the direction in which the user is looking (i.e. the position of the VR headset) and generates the appropriate images that are displayed on the small screens on the VR headset. Also, stereo sound can be generated and fed to the user's ears via the headset. Note that stereoscopic images are usually generated for the VR headset, so that the user gets the impression of visual depth when they look about the environment (for an explanation of depth perception, see http://www.vision3d.com/stereo.h tml).

Some applications of VR technology include: games (flight simulations, "Quake-like" games etc.), medicine (letting surgeons practice operations on a computerised patient), space exploration (letting operators on the Earth control robots in space or on other planets), architecture (letting architects move around computer models of the buildings they create) - the list goes on and on.

So, back to the question: How can we link a VR headset to our 360-degree video? Well, it would be necessary to digitise the video so that you could link the VR headset to the computer, and then get the computer to deliver the part of the image needed (dependent upon the direction that the user is looking in) to the screens on the headset. This shouldn't be too difficult; it's a very similar problem to that of getting the computer to show the correct direction in the 360-degree videos linked to above (e.g. the Ally McBeal set videos).

There may be a problem, though. With virtual reality, the computer generates the images based upon where in the environment the user is. But if we videotape a 360-degree environment, the user of the system will not be able move around at will within the environment, only along the path the camera took when the video was made. This is because we have no information about what other areas of the environment look like. Imagine the situation where we take a 360-degree video whilst walking down a corridor. But when the user watches the video on the VR headset, they want to go through a door off the corridor - but they can't because we didn't record this.

There are some possible solutions to this problem, but they are quite complex. For example, we could write a computer program to make a guess of what the 360-degree image would be if the user moved off the path that was originally recorded. This would not be perfect, and there is a limit to what the computer could guess - the computer couldn't make a guess based upon something it has no information about (e.g. the user couldn't "walk" through a door off a corridor if we didn't record any information behind that door). Note that, the further from the original path the user strayed, the worse the computer's guess would be.

Such a computer program could be based on a technique similar to that which enables stereoscopic vision. Using geometry, it is possible to calculate the distance from the camera (or eye etc.) of a point on an object shown in two images, if those images were captured from two different positions (rather like our eyes do). Having this information, it is possible to create a 3D model of an environment. Imagine that our 360-degree camera moves through a room. We have information from the whole room from a path through the room. So we could use information from a frame of the video when the camera was at one side of the room along with information from when the camera was at another point in the room to create a model in the computer of the room. Then, when the user moves through the room, they could stray from the pre- recorded path, and the computer could generate the appropriate image to display based on the computerised model.

Such a solution would require both sophisticated hardware and software. It is very doubtful whether a standard desktop PC would be suitable; I estimate that it would require a moderately powerful supercomputer (or many PCs networked together and running software to allow them to act as a powerful computer).

To answer your final question, about who you can trust to talk to for help with engineering, I'm not the best person to talk to. However, if you believe you have a unique idea, then it should be possible to obtain a patent on your idea. Patent law varies from country to country (and some countries do not have patent laws), but the following sites may help:

Canadian Intellectual Property Office:

http://strategis.ic.gc .ca/sc_mrksv/cipo/welcome/welcom-e.html

US Patent and Trademark Office: http://www.uspto.gov/

I am not a patent law expert, but Patents protect you from other people stealing your idea before you have a chance to exploit it yourself. You can license patents to companies so that they can use your idea by paying you a license fee.

You might also like to contact the Canadian Young Inventors' Fair Society

(http://www.rimart.com/cyif.html) who may be able to point you in the right direction.

I hope this has helped. Good luck!

Chris Rose

Imaging Science and Biomedical Engineering

University of Manchester