A/B testing is making two versions of something and running both in some way or another to see which version performs best.
For example: you could make two versions of a website, and run version A one day, B the next day and so on. Then you’re able to measure performance (e.g. how many people click on ads, or how long people stay in your site) of both versions and choose the best version.
For an app this is also perfectly possible in theory. In practice it all depends on what you want to measure and what the differences between two versions are. It should be possible, for example, to show one of two possible in-app-purchase interfaces depending on some sort of device ID, and keeping track of which interface generates more revenue.