Beat Finder / Word Finder for Music/Speech - Ever Done This?

Solar2D General Questions/Discussion
#1

Hi folks.

Every once in a while I have a client ask me if I can automatically identify the beat of a song and/or identify the point in time when words are spoken/sung.

I typically tell them:

  • These can be identified manually and stored in a table.  Then you can use that data for the actions you want, but the alignment with the song/speech may still not be perfect.
  • You should probably avoid this.
  • Corona does not have audio processing tools.

I’m tired of saying this and may just not be thinking hard enough or not thinking outside the box.

So, I’m asking you (the community at large), has anyone here done this with Corona or in some other way?

I’m open to any suggestions, including offline processing++ to do the identification and accumulation of times.

Thanks for reading!

#2

Hi Ed,

Coming from an audio background, i dont see how beat detection could be done automatically using the existing audio API.

It just cant be done (pick up the glove anyone, i dare you!)  :smiley:

#3

Hi,

I’ve done this using Flash, but that was a looong time ago. As you mention, audio processing support in Corona is too minimal for this.

-dev

#4

It is strange that you can get the duration and even seek by milliseconds, but there is no way (it appears) to get the current position of an audio stream. That could be helpful.

-dev

#5

What about off-line and/or non-Corona solutions?

While I’d love to solve it in-game I don’t hold out hope for this: no built-in audio processing and other options are too slow.

Still, I am a neophyte here so I feel I’m missing some other solution.

Again, I’m looking for any solution suggestions.  They don’t need to use Corona as long as they can be fed to Corona at some time after the processing.  I don’t need real-time processing.

#6

I haven’t delved into this subject as for audio analysis I just use an audio editor (Sound Forge is my choice and habit) for my needs, but you’d need to analyze the live wave frequencies for auto detection, something which is quite unusual for obvious reasons…its live.

There are BPM apps but the few I’ve seen just lets the user tap to detect the tempo.

Shazam and other music identifiers rely on audio fingerprints to recognize music so thats a database job.

Spotify have pre-analyzed all music and stored BPM in their analytics database.

Did a quick search and found this one for some insight.

https://nlml.github.io/neural-networks/detecting-bpm-neural-networks/

https://nlml.github.io/neural-networks/detecting-bpm-neural-networks-update/

He made data a bit more manageable by downsampling it, which reminds me, thats exactly how spotify do it before analyzing.

No point wasting processing power on irrelevant data.

11000 hz gives a playback range of approx up to 6500 hz

This covers the important frequencies for this purpose.

If possible I’d even try reducing bit resultion from 16 bit to maybe 8 bit but I have no idea if all of this can be done live on the go.

#7

Disclaimer: I didnt read the entire article above so if its off topic, dont shoot the messenger  :stuck_out_tongue:

#8

Maybe these links can help

https://github.com/allanpichardo/Unity-Beat-Detection

Or maybe corona could implement something like this

https://github.com/naudio/NAudio

As you can see, realtime audio analysis is easily done using NAudio here

https://www.youtube.com/watch?v=rj5yX0nFigg

though I think this is just too marginal and far off what most use corona for, for them to spend time on it.

#9

And this one

https://www.youtube.com/watch?v=qUlCImYOC8c

Got carried away a little but done now  :smiley:

#10

https://www.parallelcube.com/2018/03/30/beat-detection-algorithm/

Now im done!  :rolleyes:

#11

Hi Ed,

Coming from an audio background, i dont see how beat detection could be done automatically using the existing audio API.

It just cant be done (pick up the glove anyone, i dare you!)  :smiley:

#12

Hi,

I’ve done this using Flash, but that was a looong time ago. As you mention, audio processing support in Corona is too minimal for this.

-dev

#13

It is strange that you can get the duration and even seek by milliseconds, but there is no way (it appears) to get the current position of an audio stream. That could be helpful.

-dev

#14

What about off-line and/or non-Corona solutions?

While I’d love to solve it in-game I don’t hold out hope for this: no built-in audio processing and other options are too slow.

Still, I am a neophyte here so I feel I’m missing some other solution.

Again, I’m looking for any solution suggestions.  They don’t need to use Corona as long as they can be fed to Corona at some time after the processing.  I don’t need real-time processing.

#15

I haven’t delved into this subject as for audio analysis I just use an audio editor (Sound Forge is my choice and habit) for my needs, but you’d need to analyze the live wave frequencies for auto detection, something which is quite unusual for obvious reasons…its live.

There are BPM apps but the few I’ve seen just lets the user tap to detect the tempo.

Shazam and other music identifiers rely on audio fingerprints to recognize music so thats a database job.

Spotify have pre-analyzed all music and stored BPM in their analytics database.

Did a quick search and found this one for some insight.

https://nlml.github.io/neural-networks/detecting-bpm-neural-networks/

https://nlml.github.io/neural-networks/detecting-bpm-neural-networks-update/

He made data a bit more manageable by downsampling it, which reminds me, thats exactly how spotify do it before analyzing.

No point wasting processing power on irrelevant data.

11000 hz gives a playback range of approx up to 6500 hz

This covers the important frequencies for this purpose.

If possible I’d even try reducing bit resultion from 16 bit to maybe 8 bit but I have no idea if all of this can be done live on the go.

#16

Disclaimer: I didnt read the entire article above so if its off topic, dont shoot the messenger  :stuck_out_tongue:

#17

Maybe these links can help

https://github.com/allanpichardo/Unity-Beat-Detection

Or maybe corona could implement something like this

https://github.com/naudio/NAudio

As you can see, realtime audio analysis is easily done using NAudio here

https://www.youtube.com/watch?v=rj5yX0nFigg

though I think this is just too marginal and far off what most use corona for, for them to spend time on it.

#18

And this one

https://www.youtube.com/watch?v=qUlCImYOC8c

Got carried away a little but done now  :smiley:

#19

https://www.parallelcube.com/2018/03/30/beat-detection-algorithm/

Now im done!  :rolleyes: