Jev-X

Post

Thread · 8 posts

This thread explains how PostHog uses decision models to replace some LLM calls for lower cost and faster results, and where LLMs are still needed.

View 8 topics
PostHog@posthog · ↗

Building a product that requires constant LLM calls gets real expensive, real fast. Fortunately, decision models like Jev can do some of the work up to 444.6x cheaper and 193.6x faster. Here's how we use them at PostHog:

4115191.1K
PostHog@posthog · ↗

It would cost way too much to send every Replay Vision summary to an LLM to scrub personal data. Instead, we get a decision model to score session summaries, and only the ones that score below 0.4 for PII get rewritten by Gemini.

Post image
1030222
PostHog@posthog · ↗

Our PostHog Slack bot uses one to determine which model it should use to complete a task. When it receives a message, it checks a list of models to decide which one makes the most sense. Each option in the list carries a short note from us on what that model's good for, like "A general coding model. Good for most code changes and debugging" for GPT Sol.

Post image
1001174
PostHog@posthog · ↗

Decision models can also be used in combination with embeddings to more cheaply rank results than an LLM would. When a user searches for a theme like "onboarding friction" in Replay Vision, an embedding search pulls an initial list of session replays. It then passes that list to a decision model, which checks if the observation matches the initial search, then reranks them by probability. This improved ranking quality by about 25%.

Post image
101078
PostHog@posthog · ↗

Decision models aren't so good for multi-step thinking, though. Before we landed on embeddings, we tried using them to evaluate two questions about each Session Replay: 1. Is this session worth watching? 2. If so, why is it worth watching? (Choices like “error on screen,” “slow or stuck,” “user gave up", etc.) The results were awful. Processing the two questions required an hourly job that sent 12-38k tokens each and flooded the model server and block other products' calls for hours. It's better to stick to LLMs for that kind of work.

100062
PostHog@posthog · ↗

We also enhanced our deterministic SQL query chart picker in PostHog. It used to only look at rules (ex: multiple rows, a number column and a date column meant a line chart). Now, for opted-in orgs, a decision model looks at the SQL itself and first three rows of results to determine the best visualization for the data.

Post image
100060
PostHog@posthog · ↗

For open-ended jobs, LLMs should still be the default. New solutions for the middle-ground are well underway already, though. Our in-house AI research team built Jeeves for jobs that require more context than typical Jev use cases, but still need to be answered cheaply. It's a hybrid decision model that reasons before deciding when it's unsure. It's our default for decision calls at PostHog that don't name a model.

1002145
Picked for 11/19 daily picks · ranked by quality that daySee that day →
View on X ↗

Engagement counts are snapshots taken when collected.

Quoted by4 people

Only posts collected on this site, sorted by likes.