Multi-Pass Shader on camera in real-time

Solar2D Advanced Graphics
#1

Has anyone managed to get a multi-pass shader to work in real-time with a device camera?? Is it even possible??

So far I’ve got single filter effects working in real-time with my device camera, but now I need to combine duotone and grayscale. When I select this custom effect the camera just shows a black image (all the other UI is showing just a blank camera image)

Anyone know if this is even possible??

#2

Hi @RJ5,

Just to clarify, you’re attempting to apply this multi-pass filter on a live camera “fill” on iOS?

Brent

#3

Yes Brent that’s correct, I’m trying to apply the multi-pass filter to a fill on a live camera

#4

Hi @RJ5,

I haven’t experimented before using filters on the live camera feed. Have you confirmed that your multi-pass filter first works as expected on static images?

Thanks,

Brent

#5

Hi Brent

Yes I can confirm, if I use a static image my multi-pass filter works correctly

I can do a single filter on a live camera feed no problem, just can’t get multiple filters to work

Is there any other way that I can stack 2 filters on top of each other (on a live camera feed)… other than with a multi-pass filter??

#6

Hi @RJ5,

I would need to confirm with engineering, but most likely multi-pass isn’t supported on live camera feeds simply for performance reasons. Multi-pass filters typically wouldn’t tax performance if done once (on a static image), but applying them “live” at 30 or 60 FPS could potentially impact performance on the device. Is there any way you can work with the camera feed using one filter, then post-snap, you capture that image and apply the next filter to it?

Brent

#7

Hi Brent

Thanks for the help, I did think it was asking a bit too much trying to do a multi-pass filter on a live feed

Unfortunately I can’t do the method which you suggested. I need to be able to show the user a live black and white feed, but also they need to be able to edit the rgba on the fly. So combining monotone and greyscale filters would have been the perfect solution. If the greyscale filter had colour channels to which could have been edited it would have solved the problem.

I dont suppose theres another way to convert a live camera feed to black and white is there??

#8

Hi @RJ5,

For a “monotone over greyscale” effect, have you experimented with either the “colorMatrix” or “colorPolynomial” filters? One of these might give you a reasonably close effect, but I’m not overly familiar with how the coefficient table will affect your live feed.

Brent

#9

Hi Brent

Unfortunately these 2 effects will create colour images (which I’m not wanting)

I’m looking to create a similar effect to that of using coloured filters in black and white photography

In black and white photography you use different coloured filters to ensure that objects of different colours are separated and clearly defined. So for e.g. you would use a red filter to make all the blue objects in a scene increase in contrast (blue skies turn black). The final image is still in b&w, but the contrast of the blacks and the whites has shifted.

Hence my reason for wanting to combine monotone and greyscale. (works fine on static images just not on the live feed)

#10

Hi @RJ5,

Can you give me a link to (or post) some examples of what the image should look like? As in, what they’d look like for a B&W photo with a red filter applied?

Brent

#11

Hi Brent Here’s a link to a article on using coloured filters in black and white photographyhttp://www.photographymad.com/pages/view/using-coloured-filters-in-black-and-white-photography

#12

Hi @RJ5,

Thanks for the link, I’ve done quite a bit of color photography in my day (and used some color filters in some cases) but I’ve not done any B&W. I can ask the engineers if multi-pass can be added to the live camera feed, or why it might not be supported in specific.

I wonder if you might try putting the contents of the camera feed into a snapshot, then applying a multi-pass to that?

Brent

P.S. - Out of curiosity, have you tested single-pass filters on the live feed on an actual iOS device? Some filters can hit the GPU harder than others, especially if applied on an object which must update every frame (i.e. the live camera feed).

#13

Hi Brent

I grew up on B&W photography, it was easy to develop yourself and was loads cheaper than colour. 

When you say single-pass filter do you mean this method??

display.setDrawMode( "forceRender" ) local x = display.contentCenterX local y = display.contentCenterY local image = display.newRect( x, y, display.contentWidth, display.contentHeight ) image.fill = { type = "camera" } image.fill.effect = "filter.grayscale"

if so, then it works no problem at all.

Multi-pass also works no problem at all on a static image, it just won’t work on a live feed.

Unfortunately I can’t take a snapshot of the camera feed and multi-pass that as I need the user to be able to live edit the feed and view in real-time the changes that they are making. The only way I can see to do it with the snapshot method is to have it set up in an enterFrame event (taking 30/60 snapshots of the feed every second) , but surely there would be performance issues with that??

#14

You might want to test out the snapshot method, with a Runtime event essentially “invalidating” (refreshing) the snapshot every so often. But first, make sure that your multi-pass filter works as expected on a snapshot, before trying to make that snapshot update with the camera feed. :slight_smile:

#15

Hi @RJ5,

Just to clarify, you’re attempting to apply this multi-pass filter on a live camera “fill” on iOS?

Brent

#16

Yes Brent that’s correct, I’m trying to apply the multi-pass filter to a fill on a live camera

#17

Hi @RJ5,

I haven’t experimented before using filters on the live camera feed. Have you confirmed that your multi-pass filter first works as expected on static images?

Thanks,

Brent

#18

Hi Brent

Yes I can confirm, if I use a static image my multi-pass filter works correctly

I can do a single filter on a live camera feed no problem, just can’t get multiple filters to work

Is there any other way that I can stack 2 filters on top of each other (on a live camera feed)… other than with a multi-pass filter??

#19

Hi @RJ5,

I would need to confirm with engineering, but most likely multi-pass isn’t supported on live camera feeds simply for performance reasons. Multi-pass filters typically wouldn’t tax performance if done once (on a static image), but applying them “live” at 30 or 60 FPS could potentially impact performance on the device. Is there any way you can work with the camera feed using one filter, then post-snap, you capture that image and apply the next filter to it?

Brent

#20

Hi Brent

Thanks for the help, I did think it was asking a bit too much trying to do a multi-pass filter on a live feed

Unfortunately I can’t do the method which you suggested. I need to be able to show the user a live black and white feed, but also they need to be able to edit the rgba on the fly. So combining monotone and greyscale filters would have been the perfect solution. If the greyscale filter had colour channels to which could have been edited it would have solved the problem.

I dont suppose theres another way to convert a live camera feed to black and white is there??