Statsig

?????????????????? ?????? ?????????? ???????? ???? ?????????????????????????? ?????? ?????? ?????????????????????? ?????? ???? ???????? ???? ?????????????? ???? ???????????????? ???????? ???? ?????????? First, a quick primer on what “unit of randomization” means. Your “unit of randomization is the level at which you assign participants or entities to different experimental groups or conditions. Depending on your use case, this could be individuals, households, sessions, or even entire regions, depending on the design and goals of your experiment. ?? In B2C contexts, the unit of randomization is almost always the user. But in B2B contexts, there’s another unit of randomization you’ll want to use: companies. Here's a quick breakdown of each, and when to use them: 1?? ????????-?????????? ?????????????????????????? ???????? ??????: End-user features that directly impact individual experiences (e.g., UI tweaks, onboarding flows, notifications, personalization elements). This is particularly impactful for changes that are high up in your funnel, like onboarding or signup modifications. ?????? ??????????: If users share an account (think enterprise tools), it will get messy—one user sees the test, another sees control (in other words, user crossover). ???????? ???? ????????????????: Consider segmenting users by roles or usage patterns, or use a hybrid approach where individual features are randomized only for certain segments to avoid overlap. 2?? ??????????????-?????????? ?????????????????????????? ???????? ??????: Company-level features that impact shared resources and settings across users within a company (e.g., permission settings, team-wide dashboards, shared notifications). Particularly impactful when running experiments that would affect account-level metrics - like spend, number of users invited, etc. ?????? ??????????: Fewer accounts mean it takes longer to reach statistical significance. Also, accounts are very heterogeneous, so you need to make sure they're evenly distributed. ???????? ???? ????????????????: Set longer experiment durations, and use techniques to minimize variance (e.g. CUPED). Exclude outliers, or use techniques like Stratified Sampling to create balanced treatment groups. ?? Even if you’re running experiments on a company-level, you’ll want to collect metrics at the user level, which you aggregate by company. This lets you see the impact of changes on user behavior, in addition to the changes in company-level metrics.

??????’?? ?????? ???????? ???????????????????? ?????????????? ?????? ???????????????? ???? ?????????????? ??????????????????. A major challenge in B2B experimentation is dealing with small sample sizes and a skewed set of customers. The presence of whales (large customers) means that your results will be frequently skewed when a small subsample (often 10-20 customers) is randomly more heavily represented in test or control. Imagine you have two large customers. With a 50/50 split between test and control, there's a 50% chance that both of those customers get assigned to the same group, and whichever group that is will "win" the experiment if you're naively measuring aggregate metrics like revenue. Fortunately, there’s a long-established solution to this problem: Stratified sampling. Unfortunately, it’s tricky to implement. ???????? ???? ????? Stratified sampling is a method that ensures different segments are evenly represented in your experiment. When instrumented properly, it can drastically reduce variance in experiment results. We’ve created two ways to do this at Statsig - though other approaches exist and can be equally valid. #??: ?????????????????? ???????????????????? ???????????????? Our SDKs use a randomization "salt" to assign any ID type (e.g., users, accounts, orgs, etc.) to experiment groups. When you enable stratified sampling, we test multiple salts (currently 100) to find the one that best balances your groups based on a metric or attribute you select. For example - if you want to balance companies by revenue, we’ll take 100 samples, and choose the “salt” that is most balanced, so revenue is more evenly split across the treatment groups, based on historical values. Alternatively, you could have “t-shirt size” classifications for different segments of companies (i.e., S/M/L/XL), and use this to balance the treatment groups. This method, by reducing skew, reduces the spread of observed changes in AA tests (where nothing actually changed) by around 50% in our simulations. This lowers false positive rates, and ensures more consistent and trustworthy results. This gives you more confidence - in both senses of the word - in your measurement. #??: ???????????? ???????????????????????????? You can manually assign users to Control or Test groups, ensuring balance across strata. By checking the "Include Overrides in Pulse" option, manually overridden users are included in all metric analyses. This is great for your true “whale” accounts - or when there's complex logic to what strata you want to use. ????????'?? ?????? ?????????????? Cleaner, more reliable insights that better reflect your broader user base—not just the big players. Both of these are available out of the box using Statsig - which is a pretty big win for B2B companies.

? B2B Experimentation Week Day 1: How do you design B2B experiments for success? Our Head of Data, Tim Chan, is kicking off the week by highlighting key differences between B2B and B2C experimentation. Here are 3 essential considerations for running successful B2B experiments: ?? Organization-level experiments: It’s critical to ensure that users within the same organization have a consistent experience. If you’re testing a new feature or interface, all members of a specific organization should see the same version to prevent confusion. ? Connecting user metrics to their organizations: By linking individual user behaviors like login frequency or feature usage back to their respective organizations, you can understand how changes affect not just individual users but entire companies. ?? Stratification: It’s common to have customers of varying sizes, and random assignment alone might skew results. - Stratified sampling can help by balancing assignments evenly across all groups. - This ensures that test and control groups are comparable, and makes it easier to slice results across segments. Have you encountered any of these or other interesting situations in B2B experiments? Comment below and stay tuned for more B2B experimentation content throughout the week! Be sure to follow Timothy Chan, Craig Sexauer and Lin Jia for more content on org/company-level randomization, and stratified sampling.

Stoked to have finally found the analytics platform we've been looking for in Statsig! ?? After a long search, we’re thrilled to start rolling out Statsig’s powerful capabilities across all client projects. From true analytics to A/B testing and robust release management, Statsig gives us the flexibility and insights needed to drive smarter, data-backed decisions, without the enterprise price tag. Looking forward to seeing the impact this will have on our clients' growth journeys. #eCommerce #analytics

Nuestros speakers están IN. El 28 de noviembre el Centro de Convenciones Buenos Aires será el epicentro de la vanguardia en Inteligencia Artificial y tecnologías disruptivas. ?? Si buscas innovación para mejorar tu negocio, AI IN LATAM es el evento ideal. Vas a descubrir de la mano de expertos cómo la inteligencia artificial puede transformar áreas clave de tu empresa, ayudándote a bajar costos, mejorar la productividad y redefinir el futuro de tus operaciones. Meet the speakers ???Astrid Mirkin: General Manager South Cone at TikTok ???Fernando Lopez Iervasi: President Hispanic South America at Microsoft ???Pablo Moretti: VP of Product Development at Mercado Libre. ???Daniel Stilerman: Managing Director Data Science, Analytics and AI at Accenture No te quedes afuera, take a step into de future. ??? 28 de noviembre en?? Centro de Convenciones de Buenos Aires. ??? Tickets con descuento: https://lnkd.in/dA5mZWCz ????Quieres sumarte como speaker? Anótate aquí: https://lnkd.in/d4CqiT2v ecloud agency Avenida+ icomm Statsig IBM Varegos BeClever Growlat Workplace by IRSA Aivo VML infobae RESTART DDM Interaction Agency AWG SOUTH POST S.A. TraderPal #AI #InteligenciaArtificial #Ailat #B2B #Argentina

Do you work at a B2B company? Do you love driving organic product growth? Then get ready for... B2B Experimentation Week ?? At Statsig, over one third of our enterprise customers are B2B companies - including Notion, Brex, Atlassian, Figma, and many others. But B2B experimentation is rarely talked about. This is despite the fact that experimentation is a very powerful tool for B2B companies - particularly as more and more B2B companies focus on product-led growth. Next week, we'll share all the things that we've learned about how to build a successful B2B experimentation program, including: - How to get the most from small sample sizes - The type of tests you and your team should be running - How to target B2B experiments effectively - How to weed out your whale customers so they don't bias results - Guerrilla growth tactics that you and your team can run as experiments - ...and much, much more If you're interested in following along, go ahead and follow us on LinkedIn, plus people on our team who will be sharing content ?? vineeth madhusudanan, Timothy Chan, Margaret-Ann Seger, Craig Sexauer, Akin Olugbade, Liz Obermaier, Lin Jia, Pierre Estephan, Siddharth (Sid) Sharath Kumar, and Skye Scofield Happy Building! ?? ??

It’s time for another Feature Friday! Week 39: Today, we’re introducing templates for dynamic configs. You asked, we shipped ?? Our customer Figma requested the ability to create templates for dynamic configs that can be reused across teams... and now it’s here ?? Software Engineer Jairo Garciga explains how these templates can be applied at the org or team level to enable standardization. Shoutout to our customers for continuously pushing us to make the product better! Note: Templates were already available for feature gates & experiments. Learn more about dynamic configs in our docs: https://lnkd.in/gxFZhCUt

Acquired podcast's Ben Gilbert and David Rosenthal interview Guillermo Rauch—discussing how Vercel and AI will help shift time spent on devices, from consuming content to creating it.

We treat our speakers well at our Experimentation London Meetup by providing them with this special post-talk throne to relax in ?? Stewart Ehoff from RS relaxing after his superb talk on Research > Action And Stephen Situ from Forbes on stage delivering a fascinating talk on How Community Driven Content is Changing The Way We Optimise for User Behaviour Thanks to everyone from Creative CX for helping out, keeping people fed and watered! And thank you Statsig for sponsoring. The next meetup will be on the 22nd January, so signup to our group to keep an eye out for the details.

Get ready for some SaaS-y experimentation content ?? Experimentation is often discussed in a B2C context: optimizing checkouts, personalizing recommendations, and testing discounts. But what does experimentation look like in B2B? ?? Next week, we’ll explore how to build a successful experimentation program tailored for B2B companies, drawing from our experience working with leading SaaS organizations ?? Each day next week, we’ll tackle a specific topic, explore unique challenges in B2B and statistical techniques to overcome them, and share real-world examples to help you B2Be the best in experimentation ?? Follow this page throughout next week and stay tuned!