Video-to-texture plugin with audio support and everything

[devex]

Hello to all lovers of the video in videogames

So there’s been a long-standing wish for a video component that is a regular display object and follows the OpenGL display hierarchy and works on windows. This has kind of been done before, but there were always issues with audio (no audio, poor sync, etc) and general usability making them hackish to use.

I’m a firm believer of the benefits of video in videogames and so sat down one day and decided to make my dreams a reality . The result of this serious effort is the new and shiny movie plugin. Usage should be very simple with the numerous convenience functions, while still allowing you to do all the work yourself if you so wish.

While I’ve tried my best to squash bugs (so many bugs), give usage examples and make the API as easy as possible to use, I might have missed some things and I therefore welcome all contributions from the community. You could review my code and maybe enhance it, offer optimization advice, open issues and pull requests, or just proclaim how the plugin has changed your life

On a more serious note however, I’d like to kindly ask anyone with apple devices and XCode knowledge to please build the plugin for the platforms that fit in apple’s walled garden (iOS, macOS, tvOS?). Hopefully, this should be easy enough as there isn’t really any platform-specific code. If you succeed, please ping me on github, so the builds can be added to the plugin’s repository. Also let me know in case you fail and maybe I can make changes to the plugin to help you build it.

So finally as promised follows the github repo containing source code and releases for the android and win32 OSs. Also make sure to look through the README and examples in the Corona/ folder for valuable insights.

https://github.com/ANSH3LL/plugin_movie

Looking forward to your stars (on github ) and comments. Have a great day.

[Scott_Harrison]

This is so cool!

Have considered hosting this on plugins.solar2d.com?

[StarCrunch]

Hi. Sounding good!

I saw your fork appear a while ago but haven’t followed very closely, so don’t know what’s changed files-wise. I did have the Mac project building and running, then pretty much stalled. As you probably noticed when putting Android together, there are a lot of Ogg source files.

(I had some hope of adding support for RG textures (e.g. https://www.khronos.org/registry/OpenGL/extensions/EXT/EXT_texture_rg.txt) for external textures, and maybe allowing extra samplers–Solar uses up to two plus three masks, so 5, but GLES allows at least 8, desktop even more–which both have relevance to video decoding. We’ll see if this materializes.)

Also, if you’re brave, now or in the future, I think there’s opportunity for a CoronaExternalTexture-inspired audio API, maybe modeled after SDL.

It might use this, but I’ve never had a good chance to explore; it’s been forever on my ideas shelf.

[devex]

Thanks!
I have considered that, but was hoping I could have it support the 4 major platforms first. I guess if this doesn’t happen soonish, I’ll just give up and submit it to the directory anyway.

[devex]

Thanks!
Yeah, for the pull request I made, I mostly left things intact except for a few files I renamed. In this plugin however, I’ve deleted lots of unnecessary files (mostly encoding related) and tried hard to optimize for decoding. Also a few modifications to theoraplay to make audio work.

The source files are quite many especially for the vorbis and theora codecs, but I think it’s quite straightforward to build them, haven’t had any issues there. I think a good way to go about it would be to build the them as static libraries first (that’s what I did).

I was lucky enough to find OpenAL already built for win32 here.

The audio API seems like quite the challenge . Maybe I’ll look into it in the future when I have more time

[StarCrunch]

Agreed, the build itself was smooth. Just a chore to set up.

Whoops, saw the fork but not the PR. Just pulled it.

[pickerel]

A great plugin.

Thanks.

[ymmtny]

Very impressive. Working nicely with FilterViewer too.

[ymmtny]

I just built the plugin for Corona Simulator macOS. The simulator crashes at exiting (cmd+q) while playing video.

[devex]

Hi, from the screenshot provided, it seems like a segmentation fault is causing the crash. Does the crash also occur with a built project?

I’ve tried exiting the simulator while video was playing to see if I could replicate your issue on windows, but there was no crashing. Did you perhaps modify the code before compiling the plugin?

Perhaps you could share the XCode project files so I could look around for any potential problems?

This might be helpful for finding the exact cause of the crash: https://stackoverflow.com/a/66450019

[ymmtny]

I forked your plugin_movie to GitHub - kwiksher/plugin_movie at macOS

the crash is not 100% reproducible. Sometimes the simulator finishes without crashing.

I changed plugin_movie.lua to calls plugin_movieLib.dylib. For instance, there is lib.newMovieCircle(opts) in plugin_movie.lua. I don’t think this is the cause.

libogg libtheora libvorbis are from GitHub - ggcrunchy/solar2d-plugins: Source for various Solar2D plugins.

[devex]

In your xcode project, there’s references to a plugin_movie.h which doesn’t exist in my repo. I’m not sure how you ended up with that.

Also, your code is quite confusing, but I think your lib.newMovieCircle(opts) addition might be the problem somehow. I suggest building the plugin with no modifications and testing to ascertain the exact problem.

I’d recommend using the ogg, theora and vorbis code provided in the /libraries folder to build the static libraries instead of using external libraries as I made some tweaks to these libraries that I’ve provided in the repo.

[ymmtny]

Thank you for the review. Now the ogg, theora and vorbis code are provided in the /libraries folder and removed plugin_movieLib.dylib. And plugin_movie.dylib is built with your original files.

GitHub - kwiksher/plugin_movie at macOS

But I can not resolve the runtime error that symbol not found in flat namespace ‘_CoronaPluginLuaLoad_plugin_movie’, when I run it in the simulator

I don’t know why extern “C” is not working. Why?

If I rename plugin_movie.c to plugin_movie.h to include it into PluginMovie.cpp, it resolves this issue but another runtime error appears

symbol not found in flat namespace '_th_comment_clear'

plugin_movie.h was an experiment while attempting to resolve the symbol not found error. I forgot to delete it before putting into my repo. Then I thought ggcrunchy/solar2d-plugins might be doing different with macosx, so I used the ogg, theora and vorbis of solar2d-plugins, and ended up to build plugin_movieLib.dylib that was not a lua/native hybrid.

[devex]

The error symbol not found in flat namespace '_CoronaPluginLuaLoad_plugin_movie' is most likely because you aren’t building the file shared/generated/plugin_movie.c. This file is the compiled version of shared/plugin_movie.lua and must be built for the plugin to be usable. Instead of adding the entire /generated folder you should add the file directly to the build pipeline.

Regarding symbol not found in flat namespace '_th_comment_clear', I think adding the files theora.exp and theora.def to your build is what is causing the problem. They’re only needed when building with visual studio.

[StarCrunch]

@ymmtny I don’t think I did anything special with the plugin libraries. There is a theora plugin in that solar2d_plugins directory (just a much more primitive iteration of this idea), but it doesn’t do anything special, just dump all the files into a project.

As far as the crash… looking at the report, I think I have an idea:

From some other conversations, I believe you’re comfortable with the Solar source code? If you look at the array destructor and follow the logic a bit, it looks like it calls the item destructors in turn, so I suppose, if something has the misfortune of attempting to be created or updated just as you’re relaunching, you might get a double free.

Does it still happen it you go to the Renderer destructor and add, say:

fCreateQueue.Clear();
fUpdateQueue.Clear();

?

This might explain some issues I’d been having with my Bytemap plugin, as well…

[ymmtny]

Now the link errors are fixed. I updated my macos branch.

I tested it for 10 times and no crash today. I look into Renderer destructor later.

Thank you for the comments, @devex, @StarCrunch

[devex]

That’s great! With your permission, I can update my repo and add the macos build project files as well as the .dylib binaries. It might be a good idea to make a pull request.

Also, if you or anyone else is able to build the plugin for iOS, that would be appreciated.

[ymmtny]

Sure you can use it.

My commit history is not clean. Instead of a pull request, it would be better to pick the files in macos folders . XCode project files are originally from ggcrunchy/solar2d-plugins

I built it for iOS by changing the SDK maxOS to iOS in XCode settings and it worked in the XCode simulator. I have not tested it yet with my old iPad Pro. It is in my ios branch.

[devex]

Ok, I’ll look into adding the files manually, but since I can’t build them myself as I do not have access to a mac, I’m kindly asking you to share your build outputs (plugin_movie.dylib and libplugin_movie.a) so I can craft a release and finally add the whole plugin to the Solar2D plugin directory. I will credit you for the apple builds.

[ymmtny]

You can get the libraries in my ios branch.

• maxosxlibs/plugin_movie.dylib
• ioslibs/Release-iphoneos/libplugin_movie.a

It played well on iPad Pro(10.5 inch, 15.4.1).