CameraServer class

[BastiaanOlij]

This project is pretty much done at least for use on OSX and iOS and in it being used by both ARKit #24227 and ARCore #26221 implementations. Camera support on Android is thus working within ARCore.

In theory CameraFeed objects can be subclassed in GDNative so we can create GDNative based camera implementations.

There is a test project here that works on OSX and iOS:
https://github.com/BastiaanOlij/godot3_test_projects/tree/master/TestCameraServer
This sample project includes an example YCbCr -> RGB conversion shader.

There is one outstanding issue with the GLES2 renderer and that is that GLES2 doesn't support 2 channel textures required for the CbCr texture.

Seeing the difficulties I've had on Windows this platform is an excellent candidate for a GDNative implementation.
Someone else will have to do a linux implementation in due course.

[akien-mga]

X11 and Windows seem unhappy: https://travis-ci.org/godotengine/godot/jobs/268574379
You'll also have to rebase as I tweaked scene/register_scene_types.cpp a bit during a cleanup.

[BastiaanOlij]

Hey remi, Thats probably cause they don’t have any changes done to them yet.:) I’ll be implementing supporting the camera on windows soon but someone will need to help me on android an x11:)

On Sun, 27 Aug 2017 at 2:44 am, Rémi Verschelde ***@***.***> wrote: X11 and Windows seem unhappy: https://travis-ci.org/godotengine/godot/jobs/268574379 You'll also have to rebase as I tweaked scene/register_scene_types.cpp a bit during a cleanup. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2vabFGugyYXe8qDQBzQ-nIVsiF1BS8ks5scEuHgaJpZM4PDTGS> .

-- Kindest regards, Bastiaan Olij https://www.facebook.com/bastiaan.olij https://www.youtube.com/channel/UCrbLJYzJjDf2p-vJC011lYw https://github.com/BastiaanOlij

[BastiaanOlij]

Oops, I was including visual_server.cpp instead of visual_server.h, auch...

Working on Windows implementation atm, once I'm back on Mac I'll fix that clang issue.

[BastiaanOlij]

Ok the current checking is probably completely broken. I'm about to check this out on Mac, check all the blind changes I've done and rebase it to the latest master (might not get around to committing this till tomorrow).

I've been trying to get the windows version to work which has been a frustrating exercise in futility. Its not helped that my expertise on some of the approaches Microsoft seems to love (like COM ugh) is lacking.

Just for people to understand. Webcam support on windows started life as Video For Windows, which is fully deprecated, dead and buried.
They replaced it with DirectShow (part of DirectX) which is still around but key parts have been deprecated in Windows 7 and removed from Windows 10. While I'm 80% through getting this to work, the last 20% requires adding your own direct show filters written as a com DLL, not really suitable.
The current hot to trot solution is called Media Foundation, which I've found completely baffling, I don't even know where to begin.

Anyway, if someone with far more experience then me wants to have a look at it, please do. I'm going to put the finishing touches on this for Mac as I really want to get back into ARKit for which I originally wrote this.

Owh lastly, there is a nice project called libuvc which is cross platform and which we may want to use. I had issues with it not wanting to compile on Windows but with a bit of effort its possible to work around that. This library offers direct access to cameras over USB which would actually serve us well.

[BastiaanOlij]

Alright, the Mac OS X build is a working again. As mentioned before this should also work for iOS. The only thing I need to add here is discovering when new cameras are plugged in.

For all the other platforms at the very least we need to instantiate the CameraServer base class so Godot can interact with it and find out there are no cameras available. This also opens up the door to adding camera streams through GDNative.

As suggested before, libuvc may be a good open source project to evaluate to add camera support on Linux and possibly windows.

The next step for myself is adding logic to the renderer to use the camera as a background option in viewports and then I'll finish off the iOS support.

[BastiaanOlij]

Ok, this is really starting to get somewhere.

@reduz you might want to check my changes and see if I've done this in a way you agree with, I think it's pretty much in the same line with how the other background modes work. You can now select "camera feed" as the background mode in your environment settings and it will cause the nominated feed to render as the background. You do need to turn the camera on, i've not made that automatic. See the example project mentioned in the opening post which I've updated.

There are a couple of improvements that still need to be made:

• Flipping the camera image either on the X or Y axis should be made optional
• We should probably add some code to maintain aspect ratio of the camera image, right now it stretches
• I'd really like to change the code so it doesn't do the in memory copy of the camera images @reduz is this possible using the texture objects? If not possibly I should remove the support for making the textures available and just create my own GL texture objects in the camera server?
• We probably should add two sets of textures, one set currently used by the rendered, and one being updated by the camera. When the camera has updated the textures we flip a setting and discard any camera frames until the renderer pics them up. Right now it looks like there is a of locking going on as the renderer and camera are fighting over the textures.

And obviously someone will need to do Windows, Linux and Android implementations. I'll start working on iOS tomorrow and update my ARKit PR to feed ARKits camera images into this.

[BastiaanOlij]

Almost forget, one more todo, I need to react on someone plugging in/removing a webcam. Shouldn't be to hard to add but still :)

[BastiaanOlij]

The commit I just added sees a bit of refactoring in how the feeds are exposed to Godot and now completely implements adding/removing cameras on OSX so you can plug in a webcam, get a signal its available, enable the webcam, and things are nicely handled when you unplug it again.

I have also updated my sample application to show this in action:)

[BastiaanOlij]

I've just added the support to iOS, still need to debug this as it recognised the cameras but gave white output. This may have been a problem with the project files because trying out the material demo works fine.

I have also added the base camera server to all the other platforms. This means that the camera server is available but just doesn't have any cameras, prevents projects from crashing when they try to access the camera server :)

[BastiaanOlij]

Ok, one more commit for today. The iOS version of this is breathing life.

The problem on iOS is that the iPhone has high speed cameras for its slow motion feature. Basically Godot just stalled as it was getting frames faster then it could process.

This made me implement the double textures solution I wanted to do anyway. So while we're updating our camera texture the renderer uses the previous frame. After we finish updating our current camera frame but before the renderer requests using those, we'll discard camera frames. Right now we do this by ignoring them, I have to investigate whether they can be throttled somehow.

This has put the final nail in the coffin of our texture resources. Texture resources request the RID of the texture only when set assuming its contents changes, not the RID itself. I'll probably be removing that part of the solution unless someone has a smart suggestion here:)

Anyway, there is one more issue on iOS and that is that the colours are all wrong, it looks like the camera images colours are inverted. I have to research that some more.

To be continued....

[BastiaanOlij]

I did a few more tweaks and cleanups, still no further in knowing what is going wrong on iOS. On OSX the camera shows correctly, when using ARKit to handle the camera everything works fine, but getting data from the camera directly on iOS it looks like the color balance is all off:

No idea what I'm forgetting or doing incorrectly.

Note that I've found this sample project from Apple really handy:
https://developer.apple.com/library/content/samplecode/GLCameraRipple/Listings/GLCameraRipple_RippleViewController_m.html

Maybe someone can see what I'm overlooking :)

[BastiaanOlij]

After talking to @reduz yesterday, made a few optimisations by removing unnecessary allocations and reusing buffers.

One thing I'll be looking into is using yuv2rgb to do the conversion from YUV to RGB in the camera server itself so the texture uploaded is an RGBA texture. The current approach is slightly faster having the GPU do this conversion and loading slightly less data but conversion to RGB offers more flexibility.

[BastiaanOlij]

Only a small fix, I had renamed the properties at some point to properly group them but that broke the code that hides properties when not needed.

So now the camera feed id, and h/v flip properties are only shown if you've selected camera feed as the background mode

[BastiaanOlij]

@jumpy88 The ARCore functionality in Android has all the logic implemented for reading the camera as it needs that data to do all the tracking and for AR applications to display the camera feed in the background. That hasn't been merged and is waiting a bigger refactoring after some other structural changes are merged into master.

Seeing the camera server is implemented for iOS and OSX that is a good place to start. I really should find a good place to document the structure as that is indeed lacking.

In essence the camera server is a central hub where you can register the cameras that are available in the system by instantiating (subclasses of) CameraFeed objects.

When the user activates a camera the related CameraFeed instance is responsible for interacting with the camera and obtaining images which it then loads into a texture object.

Other parts of the system can grab that texture object in order to do something with those frames.

[jumpy88]

@BastiaanOlij so, what should I do is to implement CameraLinux extending CameraServer and CameraFeedLinux extending CameraFeed by following the structure I see in godot/modules/camera/camera_win.cpp (or camera_osx.mm but I'm not familiar with .mm files). Is it right?

[BastiaanOlij]

.mm files are just .cpp files but that have access to obj-c++ syntax.
I would look at camera_osx.mm and use that as a base. I never got things to work on Windows, Windows is weird.

[jumpy88]

Ok, so the syntax I don't understand should be from obj-c++ (thank you for sharing knowledge).

Windows is weird

I think it's the best sentence I've ever seen about Windows XD.
Ok, I've a lot to study about Godot module, how to extend it, how to contribute and does libuvc works (just to track, this could be useful to read data frame by frame from a usb camera). I'll make you know if I get something working.

[pwab]

Thanks Bastiaan for the clarification. I opened an issue in the docs-tracker to get this documented somehow.

Windows is weird.

I totally agree. For months I tried to implement webcam access inside a C#/WPF project (keep in mind: both MS technology). Then I got to the point that not even one library is supported anymore. The best I could do was to use Computer Vision packages like AForge (latest commit 2017) or OpenCV to get a simple texture out of a webcam stream. And I also failed there.

I now believe the camera app from the MS store uses some kind of magic. 😅

[BastiaanOlij]

@jumpy88 It just uses normal C++ syntax. It uses .mm files just so you can access some of the obj-C APIs seeing iOS using obj-C. But you can still look at it for structure.
I think I played around with libuvc but couldn't get it to work on Windows.

@pwab it's daft, I went through so many rabbit holes, from video for windows that has been totally removed, to their media classes from which camera support was removed, to their new stuff that doesn't seem to work.

Anyway we really shouldn't be having this conversation in a dead PR, not sure what a better place would be. It would be great to have others involved and do some of the other platforms as I'm knee deep into VR stuff atm.

[subjectdenied]

i have a 3.2 fork with a working webcam implementation for linux using v4l2, kinda hacked together using various sources (and based on the osx implementation), but working. if someone is interested in stabilizing it, i'd love to see this functionality in godot

[subjectdenied]

@pwab did you look into the artoolkit5 or artoolkitx sources? afaik cam works there in windows
https://github.com/artoolkitx/artoolkitx

[choonway]

@BastiaanOlij have a working webcam implementation based on microsoft's media foundation API - #44498

[jumpy88]

@BastiaanOlij I've two proofs of concept showing how cameras can be acceded in Linux. The first one is based on OpenCV and the second one on UVC.
https://github.com/jumpy88/godot_ocv-camera-reader
https://github.com/jumpy88/godot_uvc-camera-reader
I'm also planning to do the same with v4l2.
Before to start deeping into Godot source code I'm posting this here to discuss which would be the best solution to be included in Godot. I think v4l2 would be the best one not requiring to link against external libraries, but let me know your opinion.

[Calinou]

Before to start deeping into Godot source code I'm posting this here to discuss which would be the best solution to be included in Godot. I think v4l2 would be the best one not requiring to link against external libraries, but let me know your opinion.

Indeed, I think using v4l2 would be the way to go here. OpenCV is a huge dependency, so we'd rather not add it to core.

[jumpy88]

Yeah, I know OpenCV is too large, I hoped UVC was small enough (because I had some already working code based on it). Let's see if tomorrow Santa Claus will bring us a Godot + v4l2 sample.

[jumpy88]

Sorry for the delay. Santa Claus had some problems to travel due to covid restrictions. Anyway, here we are:
https://github.com/jumpy88/godot_v4l2-camera-reader

[jumpy88]

Hello everyone, I'm trying to get the same result of the previously linked repositories on Android. My experiments are based on https://github.com/komakai/UVCCamera/tree/komakai-fixes. I can build and run UVCCamera samples but can't figure out how to use its libraries (.so files you can find in libuvccamera/src/main/libs after building it) directly in C++ (without using provided Java wrappers). I think the problem is I've to use the modified libuvc functions taking int fd additional parameter but I've no idea how to get it. I've created a dedicated repository but at the moment it fails setting up the camera
https://github.com/jumpy88/godot_uvc-camera-reader_android
Any help to reverse engineer UVCCamera (provided with almost no documentation) would be very appreciated. Thank you in advance to everyone contributing to this effort.

EDIT: I've created an issue in UVCCamera repository to discuss about this saki4510t/UVCCamera#593

[jumpy88]

Sorry for the delay. Santa Claus had some problems to travel due to covid restrictions. Anyway, here we are:
https://github.com/jumpy88/godot_v4l2-camera-reader

I see there is some interest in V4L2 proof of concept so I'm starting studying camera_win.cpp by @choonway to integrate my sample code in a camera_linux.cpp. I'm not sure to understand the logic of CameraServer and CameraFeed: will my code run on a different thread and should write the new frame to a buffer when it is available (as in a producer/consumer pattern) or should it ask the device for the new frame and return it when asked by something else?

[jumpy88]

Update:
my proof of concept
https://github.com/jumpy88/godot_ocv-camera-reader
thanks to @komakai work (see komakai/opencv@2e2df44 commit) now also support built-in android cameras (it doesn't work with external USB cameras).

[Schmetzler]

I do not know much about Windows development, but maybe the implementation from OpenCV could be helpful for using the MediaFoundation Framework.

https://github.com/opencv/opencv/blob/master/modules/videoio/src/cap_msmf.cpp

And I was wondering, if we use V4L2 for Linux does that mean that the godot itself must be distributed under the GPL as V4L2 is GPL? Or can it be some kind of plugin... but if used in a game or tool it becomes GPL too?

[jumpy88]

@Schmetzler OpenCV would be useful for many aspects, but is a too extended library to be directly included in Godot. The best option would be to create a plugin to import its functionalities when they are required. I published my POC using it to inspire someone about this, but it's a task too large so I don't manage to do it alone. If you are interested in integrating OpenCV in Godot we can cooperate to create this plugin.

I'm not expert about licences, but I think you can call a function from a GPL library in your code with a different licence, otherwise no software but GPL could be executed in a Linux environment.

[Calinou]

I'm not expert about licences, but I think you can call a function from a GPL library in your code with a different licence, otherwise no software but GPL could be executed in a Linux environment.

You can't, as you have to link to the library to be able to call functions from it: https://www.gnu.org/licenses/gpl-faq.html#GPLStaticVsDynamic
dlopen() is still considered linking, even if it's dynamic.

GCC has a runtime library exception so that proprietary programs can be legally distributed, but this isn't the case for most GPL-licensed software and libraries.

Either way, I recommend doing some online searching to see if V4L2 has already been successfully used in proprietary software.

[Schmetzler]

I found out that there is libv4l which is distributed under LGPL, which is not that problematic.
My guess is that v4l2 itself seems like a driver for different camera devices and libv4l is a tool to grab images from these cameras.
With this it should be possible to create a CameraFeed for Linux (it just adds a dependency to v4l-utils which is installed in most distros by default).

I would like to try to implement this.

[subjectdenied]

what about gstreamer? is should work on linux, windows, mac and also would allow more than "just" capturing a webcam (like webrtc, ...)

[benjarmstrong]

I just made PR #49763 which is a feature complete Windows implementation including:

• Hot-plugging support
• YUV->RGB via the CameraFeed shaders (easily decodes 4K input)
• Compiles via MinGW

I understand people were already working on this, but I needed a working solution ASAP. I've recently been contracted to build a Windows multimedia app that requires camera input and I figured that while I'm dealing with the MediaFoundation API anyway I may as well build it into Godot which is very well suited to my project.

Any feedback on the implementation would be appreciated.

[georgewsinger]

i have a 3.2 fork with a working webcam implementation for linux using v4l2, kinda hacked together using various sources (and based on the osx implementation), but working. if someone is interested in stabilizing it, i'd love to see this functionality in godot

@subjectdenied Do you have this fork public somewhere? I checked your repos and didn't see it.

[Calinou]

@georgewsinger This seems to be #47967.

[JA-Developer]

Hello. What about FFMPEG?. It supports Linux, Mac and Windows. It's license LGPL.