A Product Data Scientist’s Take on LinkedIn Games After 500 Days of Play

I-day streak on LinkedIn Games. Sure, LinkedIn additionally has video games, they usually’ve been round for over a yr. On occasion, I discover new video games, design tweaks, and new options being rolled out. As a Knowledge Scientist, I’ve all the time questioned what LinkedIn is making an attempt to attain with LinkedIn Video games and the way they’re testing the modifications. 

With AI augmenting and even automating many coding and fundamental analytics duties, product sense and area experience grow to be increasingly necessary for knowledge scientists. Due to this fact, on this article, I’m utilizing LinkedIn Video games for instance to indicate how a Product Knowledge Scientist thinks and works. That is additionally the kind of psychological train I observe when getting ready for product case interviews.

I. What’s the purpose of LinkedIn Video games

Step one in any product case is to know the product’s purpose. Per LinkedIn, “Video games on LinkedIn are thinking-oriented video games that will help you sharpen your minds, take a fast break, and have the chance to attach with one another and spark conversations.”

These video games are fast mind teasers, in order that they do assist customers “sharpen minds and take a fast break” to some extent. However I imagine the actual intention hides behind the final half — “join with one another and spark conversations”. 

Why does this matter? LinkedIn generates most of its income from expertise options, promoting platform, and premium subscriptions. All of those depend on an energetic consumer base — recruiters want a big pool of energetic candidates, advertisers want focused impressions, and the worth of premium subscriptions will increase with the community measurement. Furthermore, the important thing to sustaining an energetic consumer base is consumer engagement and interactions, which finally result in increased retention.  

In knowledge language, MAU (month-to-month energetic customers) is likely one of the commonest metrics to measure the energetic consumer base of a product. MAU in month X = energetic customers in month X-1 + customers acquired/resurrected in month X - customers churned in month X. For LinkedIn, I imagine LinkedIn Video games is a function that’s designed to develop MAU by decreasing the final part, “customers churned this month.”

II. How does LinkedIn Video games obtain this purpose

Now that we’re clear in regards to the purpose of bettering retention, the subsequent query is, how does LinkedIn Video games obtain it? I feel there are two mechanisms — direct interactions with LinkedIn Video games and oblique engagement pushed by coming again to the platform. 

1. Direct interactions with LinkedIn Video games

Each day, LinkedIn publishes a sport publish and encourages customers to share their scores and suggestions. That is precisely what they meant by “allow you to … join with one another and spark conversations.” Beneath is a screenshot I took at round 10 PM Pacific Time on 11/29 — round 22 hours after the every day Zip sport was refreshed. Yow will discover the entry level to this publish after ending a sport, or it would present up in your homepage. This publish had 1240 reactions and 1370 feedback. Many customers publish their scores and work together with one another. 

This type of social interplay is valued by many LinkedIn customers. Sharing your good sport rating can be just like sharing a small achievement, so it doesn’t work towards the skilled social community picture of LinkedIn. Because of this, LinkedIn Video games creates a community impact that will increase retention. 

2. Oblique engagement from returning to the platform

In the meantime, there are completely folks like me who’re merely hooked on the video games however by no means share scores or touch upon the publish. No interplay doesn’t imply LinkedIn Video games doesn’t obtain the retention purpose for this group of customers. The truth that it brings customers again each day is already a robust retention lever. 

Linked Video games achieves this by making a behavior loop. Let me put it within the Hooked Mannequin (Set off-> Motion -> Reward -> Funding) to unpack it:

1. Set off: Customers are prompted to return by exterior triggers like push notifications and homepage modules, and inner triggers equivalent to the will to take care of a streak.
2. Motion: The puzzles are simple to know, low-friction to enter, and fast to play.
3. Reward: Customers get a distinct puzzle each day, earn streak badges, and might compete with their connections.
4. Funding: Customers “make investments” by constructing a streak, getting connections to play, sharing outcomes, bettering their leaderboard rank, and so on. Each day, customers construct up sunk effort, making it tougher to cease.

With this behavior loop, customers come again every day. So long as a consumer opens LinkedIn, there’s additionally an opportunity that they are going to try different issues, like community updates, messages, job openings, and so on. These actions might all result in significant engagements outdoors of the Video games function and enhance total retention. 

III. Experimentations on LinkedIn Video games 

We coated the purpose of LinkedIn Video games and the mechanisms behind it— LinkedIn Video games goals to enhance consumer retention by encouraging interplay on Video games content material and growing total product engagement. As a knowledge scientist, when you work on this product, a key a part of your job might be collaborating with Product Managers, Designers, and Engineers to brainstorm initiatives and run experiments to measure the retention influence. And that is clearly occurring with LinkedIn Video games, as I’ve seen so many design modifications over time. Let me stroll by some examples and focus on how knowledge scientists might be concerned.

1. Entry factors to LinkedIn Video games

Proper now, you’ll be able to entry LinkedIn Video games by:

• Games hub
• Seek for video games on the LinkedIn Search bar
• My Community web page
• At present’s Video games part below LinkedIn Information in your Desktop homepage or Aspect panel within the LinkedIn cell app
• Notifications

However this isn’t all the time the case. I keep in mind sooner or later the entry level on the My Community web page disappeared, and I needed to search within the app to seek out the video games. However a number of days later, it appeared once more. The placement of entry factors determines how simple it’s to seek out the function, for each new and returning customers. However extra entry factors aren’t all the time higher. Whereas extra entry factors enhance visibility, every of them can create a contextual bias — customers who land on My Community may behave in a different way than those that come by a notification— thus, completely different entry level has completely different impacts on engagement and retention. In different phrases, they are going to cannibalize one another. 

For instance, the My Community entry level sits under the invites and above the connection suggestions. When a consumer visits this web page to play the every day sport, inevitably, they are going to see the pending invites, and it’ll remind them to take motion — increasing connections is a vital a part of making a consumer’s LinkedIn expertise significant and precious. In the meantime, in the event that they go to their homepage for the video games, they are going to as a substitute see different customers’ posts, and usually tend to work together with the posts.

Several types of interactions have completely different impacts on retention, and it’s laborious to estimate the precise influence of eradicating/including an entry level with out operating an experiment. Now the duty is on the info scientists to design the experiment. 

Right here is the way it might look:

• Experiment design: management = present design, therapy = eradicating the entry level on My Community
• Randomization unit: user-level A/B testing, 50% customers will randomly see the management vs. the therapy design
• Main metric: 7-day retention price — time window can range primarily based on how shortly we need to measure the retention influence and any previous learnings. One caveat is that retention is a lagging metric, and LinkedIn Video games may need comparatively low site visitors in comparison with the remainder of the platform, which makes it tough to detect retention influence within the brief time period. In that case, the first metric may have to shift to a number one indicator of retention, or knowledge scientists may have to depend on causal inference methods to estimate the retention carry extra reliably.
• Secondary metrics: % customers performed a sport; % customers interacted with community posts; % customers added connections; Common periods per consumer; Common time spent per consumer
• Guardrail metric: common app/web site efficiency

Knowledge scientists will work with the cross-functional crew to align on metrics primarily based on the purpose of the experiment, run energy evaluation to find out the experiment time size and scope, conduct implementation checks, and ultimately analyze the outcomes to make a name on the most effective mixture of entry factors. 

2. Notifications

A number of months again, I began receiving reminder notifications like “You’re on a xxx-day streak. Play xxx now to maintain it going”. Later, after ending the video games, there’s one other set of notifications saying “congrats on ending xxx”. 

Notifications could be annoying, however they’re very efficient in bringing customers again. For instance, Duolingo is known for its artistic and “psychologically manipulative” notifications (I’m on a 1735-day Duolingo streak by the way in which). Their early blog post described how Duolingo used multi-armed bandits to seek out the best-performing notification.

Equally, optimizing notifications can have a huge effect on LinkedIn Video games. Knowledge scientists can run experiments to check:

1. When to ship the reminder notification — it might be throughout lunch break or after work time when customers usually tend to be out there, or when the consumer often opens the app, and even after they performed the sport yesterday.
2. When to ship the congrats notification — the congrats notification might be used to carry a consumer again to the app and encourage them to publish their outcomes and work together with different gamers. Equally, we will take a look at sending it proper after ending the sport, or perhaps later within the day when extra customers have performed the sport.
3. The message textual content — ought to the tone be impartial or extra aggressive? How lengthy ought to or not it’s?
4. The CTA (name to motion) textual content — “Clear up now”? “Play now”? “Lengthen your streak”? Totally different textual content on the button might result in completely different click-through charges.
5. Frequency — if a consumer doesn’t come again to play the sport after the primary notification, ought to we ship one other reminder?

Right here is only a brief record off the highest of my head, but it surely’s already a lot of completely different combos of notification designs. It’s completely potential that textual content A coupled with timing X is best than textual content B coupled with timing Y. Due to this fact, operating experiments for every resolution one after the other is each inefficient and will result in a sub-optimal end result. That’s the reason Duolingo talked about the multi-armed bandit framework above. It’s a framework to check a number of variations concurrently, and in contrast to conventional A/B assessments, it hastens the experiments by mechanically diverting extra site visitors to the profitable arms primarily based on a reward perform and decreasing the variety of arms within the take a look at shortly. Due to this fact, the multi-armed bandit might be very helpful to check LinkedIn Video games notifications. If you wish to study extra, here is one other article by Sew Repair on how they use multi-armed bandits of their experimentation platform.

So what’s the info scientist’s position right here? After all, they are going to brainstorm with the stakeholders to provide you with completely different variations, outline the reward features (e.g., whether or not a consumer performs in the present day’s puzzle), run the multi-armed bandit setup, and interpret the outcomes. 

3. Sport outcomes web page 

One other space the place I’ve seen many modifications is the sport outcomes web page. After ending the sport, the consumer first lands on a outcomes abstract, with enticing stats playing cards like “On hearth 500-day win streak!”, “Prime 95% All gamers”, and “Smarter than 90% of CEOs”. It additionally has a distinguished “Share” button that prompts you to share your outcomes as a publish or as a direct message to your connections.

After that, there’s a lengthy outcomes web page with seven main sections:

1. Header — how shortly the consumer solved the puzzle with copy and share CTAs.
2. Connection leaderboard — the place you rank amongst your connections. Should you click on on “see full leaderboard”, there are CTAs to nudge connections who haven’t performed in the present day.
3. The “play one other sport” CTA asks you to discover completely different video games.
4. One other abstract panel with extra stats, together with all-time win price, finest rating, streak badges, and a push notification toggle.
5. Weekly trade and college leaderboards with share choices.
6. Hyperlink to the every day sport publish, the place you’ll be able to react or remark immediately.
7. “Unlock this week’s bonus puzzle” by inviting your connections to play the sport. 

Do you see the sample? Each part has some CTAs to encourage sharing, engagement, or social interplay. Nevertheless, is the present sequence of playing cards the most effective sequence for the retention end result? Is there a greater UI for the outcomes stat playing cards with the next share price? Do folks even care in regards to the rating of their employer and college? 

To reply these questions, a knowledge scientist would design experiments just like the one we mentioned within the entry factors part to measure the click-through charges, interactions, and total retention influence.

To take one step additional, completely different customers may need completely different preferences. For instance, 

• Person A needs to “exhibit” their rating and the way a lot smarter they’re than the CEOs, so the present sequence works completely for them, as they’ll do it proper on the primary display screen. 
• Person B feels a robust belonging to their neighborhood, so they are going to share the leaderboard, asking coworkers or classmates to affix the sport to enhance the rating of their firm or faculty. Due to this fact, displaying the leaderboards on high will enhance their engagement. 
• Person C likes to share their puzzle suggestions and focus on with different gamers, then we should always transfer the publish up on the outcomes web page. 
• Person D merely enjoys the sport, and they might invite others to unlock extra puzzles if they’re supplied this selection. However with the present design, they may not even scroll down all the way in which to the underside and miss the “unlock this week’s bonus puzzle” card. 

This type of customized outcomes web page makes a number of sense theoretically, however make it work is one other difficult knowledge science query. Knowledge scientists might section customers primarily based on consumer profiles and previous actions — for instance, what number of connections they’ve, what number of of their connections have performed the video games, if the consumer usually posts or feedback, and so on. Then the info scientists might analyze the experiment outcomes by completely different consumer segments to determine which design works the most effective for every section and provide you with the personalization technique. To make the system even smarter, knowledge scientists might construct a machine studying mannequin to foretell the cardboard format that maximizes engagement for every consumer. 

4. Nuance: Community impact

Final however not least, let me discuss an necessary nuance for operating experiments on a social platform like LinkedIn — the community impact. A/B testing has a robust Secure Unit Remedy Worth Assumption (SUTVA), which assumes a person consumer’s end result is decided solely by the therapy they obtain, and isn’t affected by the therapies of different customers. Nevertheless, this doesn’t all the time maintain on social networks. 

Contemplating the LinkedIn Video games instance — Assume we modified the leaderboard UI and, in consequence, customers within the therapy group have the next probability to “nudge” their connections. And lots of “nudged” customers — some within the management group — find yourself enjoying the sport too. This community impact biases the experiment end result and dilutes the relative influence between therapy and management. LinkedIn has written about this actual problem and walked by how they detected the influence of this interference utilizing cluster-based experiments. Briefly, LinkedIn teams closely-connected customers into one cluster whereas minimizing interactions between clusters, then they use clusters because the randomization unit (customers in the identical cluster both all go to therapy or all go to regulate) to measure the influence with minimal interference. Due to this fact, for modifications that might have a robust community impact, the cluster-based experiment is an effective various. Although the trade-off is that cluster-based experiments usually scale back energy, in order that they require cautious cluster building and energy evaluation.

LinkedIn Video games appear easy, however behind each button, notification, streak badge, and leaderboard, there’s probably a series of product hypotheses, experiments, choices, and knowledge science work. 

After all, that is simply my mind train as a LinkedIn consumer, however I hope this text helps you higher perceive what Product Knowledge Science seems to be like in observe. When getting ready for product case interviews and even once you discover a brand new function on a product you utilize frequently, you’ll be able to apply comparable psychological workouts to sharpen your product sense and grow to be a stronger knowledge scientist.