To begin to answer this question, I'd break ASO down into:
- Conversion rate optimization: finding the screenshot & asset combination that maximizes conversion rates. Once you estimate the improvement in conversion rate(via either testing in ASA or third party tools) and your organic baseline, the incremental improvement in organics should be easy-ish to calculate.
- Keyword/metadata optimization: configuring keywords and metadata so as to improve your ranking for certain keywords. This is relatively easy at early-ish stages of an app -> I've known of apps going from almost zero to hundreds of daily installs fairly quickly based on keyword/metadata optimization. It's when you're at significant scale that it becomes harder to quantify this. The primary way I'd do this is to establish an organic baseline(and make sure this is done by geo) - and do a pre-post analysis while keeping paid marketing activity roughly constant. In practice here is how this would look:
-> Look at average daily organics for a month prior to ASO activity.
-> Keep paid installs roughly constant.
-> Look at average daily organics for a month after ASO activity - compare with the prior period.
I will also add that once you're at a significant scale, keyword/metadata ASO becomes less impactful. I've had keyword/metadata ASO activity done on a couple of casual games that were getting 2000-3000 organic installs a day in the US -> and even though we saw improvements in keyword rankings, we saw no meaningful movement in our total organics. This leads me to believe that unless you're in a super-mainstream category(eg - "weather"), keyword optimization does not have massive potential to move the needle(I'd estimate no more than low triple digits in daily installs). [I would of course love to see other folks' data around this].