I suspect graphically this is by and large what Worms did. (I’m more doubtful with Scorched Earth.) On the collision side of things I imagine they were maintaining a quadtree or k-d tree, or something like them. (This information might very well be out there.) This would allow for sparsity (back in an era of almost unthinkable RAM budgets :D), provide for collision with fairly simple response needs, and also permit the AI to do “speculative” collision (line-of-sight tests, checking trajectories for a clear shot, area effects, etc.), while keeping the search neighborhood manageable. But this is all a guess.
As far as “other experience”, my own (unfortunately probably cryptic) response on the show was based on a sort-of-related technique I’d developed, either before the Porter-Duff stuff hit or at least when Pro wasn’t yet free (and certainly before shaders landed, to say nothing of the recent texture canvas resource). The basic idea was to assign each tile an 16-bit pattern, with those bits indicating which parts of the tile were visible. I baked these, in the form of 4x4 black-or-white blocks (one block per bit), into a mask assigned to every tile, centering it in each case to align with the appropriate “mask tile” at the moment.
4x4 was the largest square power-of-2 shape that was sane: there are 2^16 - 1 (just shy of 64K) possibilities, but by eliminating isolated blocks or 1-block-thick filaments (something in this spirit), it can be reduced to 235 or thereabouts. The offending tiles are simply rehashed to the nearest match, i.e. with any stray block / filament removed.
(The implementation is a bit scattered, but let me know if you’d care to know more.)
64K also happens to be the guaranteed integer range in vertex shaders, so I have since translated the above to a shader technique as well (the reduction step is no longer necessary but could be kept for aesthetics), which I can drop on a tile’s fill instead. (The mask approach is a dog to maintain, so I might just retire it.) That range also means that I can flag the 16 neighbors, which allows for e.g. doing a soft fade on edge pixels. This is still a bit unruly, so at some point I ought to throw a friendlier interface on top of it.
I maintained collision slightly differently (array of booleans), but the information content was basically the same.
That said, I don’t know that this is better than the blend mode approach, if that suffices. As I said, this was implemented when that wasn’t conveniently at hand. (Uff, I’d be sad if all the work was for nothing…) I haven’t explored how well the latter allows “repair”, say if you want to restore parts that have been masked out, which was an original need of mine. It might very well be trivial.