VentureSwell

Keep buying ads, nuckers.

Loading mentions Retweet

With JobSyndicate, we're essentially running a high-payout distributed affiliate program. As we build out our internal analytics platform, we're finally getting to a place where we can begin split testing a wide variety of changes to optimize our publisher network with the first (of endless) iterations.

There are several component metrics that we're keeping a particularly close eye on: CTRs (of course), application rates, application ratings (employers rate applications to track channel performance, among other things), and conversions.

In the long run, conversions are the only thing we care about - employers need to grow their teams, and publishers need to pay their bills. We're big believers in the importance of building a network in which the risks and rewards are justly distributed, and we consider shouldering the optimization burden to be a long term strategic advantage. By choice, we're the ones with the greatest incentive to drive performance. The hiring process can be a long one, though, and so conversion rates will often be low and volatile; both are conditions that make ad optimization difficult.  In our case, it's useful to focus on CTRs, app rates, and app ratings as proxies for conversions.

Right now, it looks like we'll address them fairly sequentially (though not necessarily linearly): drive CTRs, then app rates, then ratings. We recognize, though, that optimizing each component separately can lead to classic sub-optimization, in which overall performance suffers. Will we improve CTRs while alienating the users that are actually more likely to apply, and will increasing apps/click filter out candidates most likely to convert? It's definitely possible.

As with many startups, though, one of our biggest weaknesses is a lack of data. By focusing on the lowest hanging fruit (click-throughs), not only are we directly optimizing the initial application process, but we're also increasing the amount of data we collect. More clicks leads to more opportunities to apply, and so on down the funnel. This won't be a completely linear process, either. Nivi pointed out that we'll probably run out of ideas to A/B test click rates with, for example, and move to improving the application process, and then return again (hundreds of times) to the initial interaction point.*

What do you think? I think my bigger point is that, especially in startups, you may often have to run the risk of sub-optimization in situations like these because you don't have the data not to. We believe in building to learn, and it could be that the marginal data (via data collection growth) we'll get as we sub-optimize is worth both the risk and the explicit costs themselves.


*It could also be that it's not even worth it to split test new optimization improvements at this point, that we should just make all the improvements we can as quickly as possible. Knowing what leads to improvements can obviously be hugely valuable, but then so can improving performance quickly enough that you stay alive. Prioritizing possible optimization features is a whole 'nother story, too.

Loading mentions Retweet