If I remember correctly, App Store imposes a limit of 150MB for downloading apps over a cellular connection. If your app is larger than this, then the user will be prompted that they can only download the app over a wifi connection. I think Google Play has, or used to have, a similar limit of 100MB. It’s been a while since I read up on these, so I might be wrong.
Other than that, it’s as Rob said. The smaller the app is, the faster your players can get to it and the less of a hassle downloading and keeping the app is for them. That being said, I have several games on my devices are several hundreds of megabytes or a couple of gigabytes and for those games, the sizes are alright.
I would recommend that you create the game that you want to create and avoid worrying about the size of the game during production. If you have to, it is quite easy to cut down the size of the game in post production by applying (near) lossless compression to your assets, both images and audio.