I want to do some multivariate testing on a web page. But before I do so, I’d like to have a brief idea of the math behind it. Anyone able to tell me what to read on, or point to a resource, or give me a description?
I’m guessing, without having done any reading, that I’m looking at various results - or perhaps one ‘success/fail’ result from a bunch of web pages with defined variations, then starting with a hypothesis that ‘change A’ on any of those pages had no impact on the ‘success’ of that page. But I could be whacked even thinking that way, I dunno.
Pointers appreciated.
what kinda data do you have?
you also need to define what success is, like objectively. Because you need a “label” for it.
Some of this depends on what you’ve already know.
If i am understanding you correctly, you are talking about so-called “A/B” testing, but with more than just two options?
I know of at least two areas to know related to this.
The first is sometimes called “experimental design”, and was really invented by fisher when he worked in agriculture during the early 20th century.
The idea is this: if you are varying multiple things- for example, a kind of seed, a location, and a fertilizer type- then you want to vary multiple things at once, in a way where you have equal numbers of everything (“balanced”.) Do this cleverly, and you reduce the number of experiments you must do, and therefore cost. I don’t know much about that since most of the data i work with is “observational”, and that’s probably true for most or all in this forum.
the other area to know is how the comparison is actually done after the experiment is designed. usually that is some variation of a linear statistical model. but i thought i had seen posts in the past from you where you have worked with this already. maybe my memory is off:
That’s correct.
are you familiar with linear models with multiple variables?
It’s been a while, but I did some stuff in my undergrad.
I just found this, I hope its what I need:
That looks like a good choice to me. I’d definitely stick with something very focused on that particular application, particularly if you haven’t looked at this kind of model in a long time.
If that book is too technical i wouldn’t be afraid to go less technical.
In general, i think “domain knowledge”, or knowledge about what you are modeling, can make up for a lack of statistical knowledge.
Thanks for the input, I appreciate it.
I should be fine with the book. I’m expecting it’ll be a refresher on the regression testing course I took.