#10: Bandit Testing

I knew that bandit testing could be an alternative to A/B testing. But I didn't know when it would be more suitable than A/B testing. So, I did a deep dive, and that's the main story this week.

Bandit testing comes from the multi-armed bandit problem. How does bandit testing differ from A/B testing?
In A/B tests, I split the traffic between a few variants, and I let the test run for a fixed period. After the fixed test period is over, I find the best variant and direct all the traffic to the best variant.
However, I don't have to wait until the test period is over in bandit testing to utilize the information I already have. It helps me to optimize for opportunity costs.

So when is it more suitable to do bandit testing? Really short and long tests.

• There isn't enough time to let the A/B tests finish for small tests. For example, newspaper headlines, campaigns, and promotions are relevant for a short period. If I wait around to find the best variant with the A/B test, the campaign might already be over by the time I find the best variant or the news cycle might have already changed.
• For long-term tests where I want to do continuous optimizations, which is not possible with A/B tests.

I found the blog post When to Run Bandit Tests Instead of A/B/n Tests, helpful in understanding bandit testing. Give it a read to get a more in-depth explanation.

🐍 Caching Results Of Instance Methods

I was looking into lru_cache in python, which led me to the blog post - Don't Wrap Instance Methods with functools.lru_cache Decorator in Python. Adding lru_cache decorator to an instance method is not a good way to cache the results. Adding it to static and class method is fine, though.

When you decorate an instance method with functools.lru_cache decorator, the instances of the class encapsulating that method never get garbage collected within the lifetime of the process holding them.

📚 My Reading & Listening List

I compiled the list of the newsletters and the podcasts I read and listen to regularly. I also included the books I'm reading currently and the ones I've finished reading. Check out my list, and maybe you'll find something you want to read.

🗓 Last Week, Next Week

Last week, I got to work with Jsonnet for the first time. Jsonnet is a templating language, and I'm using it to automatically generate the Kubernetes manifests.

This Saturday, I went to T.Y. HARBOR, a waterfront brewery in Shinagawa. I was visiting a brewery after a long time. They had an excellent collection of craft beers, and I loved their pale ale and IPA. Craft beers and NY-style chicken wings were a great way to spend my Saturday.

I also finished watching the Ozark season 4 part 1 but didn't like the story that much. Julia Garner's performance stands out the most. Jason Bateman continues to have great comedic timing and delivers some of the best one-liners in the show with a straight face.

I'm late for my own felony. See you later.

Next week, I want to finish reading the book On Writing Well. I'm about 75% done now and will complete the rest.

Cheers! Have a productive day.
Prashant Anand (@primaprashant)