Job description
AI Product Engineer - ClickStack
About ClickHouse
Recognized on the 2025 Forbes Cloud 100 list, ClickHouse is one of the most innovative and fast-growing private cloud companies. With more than 3,000 customers and ARR that has grown over 250 percent year over year, ClickHouse leads the market in real-time analytics, data warehousing, observability, and AI workloads.
Job Summary
We're on a mission to transform how companies use data. Join us in building the AI layer for Observability! ClickStack is the open-source observability platform we're building at ClickHouse — logs, metrics, traces, and session replays unified so engineers can find root causes quickly. The interesting work now is in the agent layer: systems that can investigate an incident at 2 AM, propose a root cause, and hand the on-call a concise summary by the time they've logged in. We're hiring an AI Product Engineer to build agentic capabilities on top of a petabyte-scale observability platform, with a focus on developer experience.
Key Responsibilities
- Build agents that investigate incidents. They surface anomalies, answer "why is production broken?", and use ClickStack as their substrate.
- Write skills, not just prompts. Build a library of reusable skills that captures how our team debugs, finds root causes, writes ClickHouse queries, and runs incident response, so agents pick up the right playbook instead of starting from scratch.
- Own the agent stack end-to-end. Context engineering, tool design, evals, tracing, cost. You're responsible for whether the agent works in production.
- Make ClickStack a great place to run AI workloads. Build the MCP servers, SDKs, and integrations that let customers' agents read telemetry, take action, and stay observable themselves.
- Work in the open. Collaborate with OSS contributors and customers, debug their problems alongside them, and feed what you learn back into the product.
- Tackle the hard parts. Latency, cost, context window limits, eval coverage, hallucinations on real telemetry.
Requirements / Qualifications
- 5+ years of software engineering experience, including 1–2 years on LLM-powered systems or agents in production.
- Strong backend skills in TypeScript/Node.js and/or Python. Comfortable in both, even if one is primary.
- Hands-on experience building agents: multi-step tool use, planning, memory, error recovery.
- Experience designing skills (Markdown-based workflow encodings, Anthropic-style or similar) and a clear view on when a skill, a tool, or both is the right fit.
- Experience with MCP: building servers, designing tools, and thinking through auth, scoping, and observability for agentic systems.
- Strong evals practice: golden sets, LLM-as-judge, regression detection.
- SQL proficiency — you can write ClickHouse queries directly.
- Comfort with Docker and Kubernetes.
- Active in open source and the developer community.
Bonus Qualifications
- Built or operated production agents in observability, incident response, or SRE.
- Strong opinions on agent observability — tracing, cost attribution, eval pipelines, OpenTelemetry for agents — and ideas on how to improve it.
- Experience with prompt caching, context compaction, or other techniques relevant to running agents on production telemetry volumes.
- Experience with columnar databases and event ingestion pipelines.
- Contributed to or maintained an open source AI/agent project.
- Familiarity with Go, Rust, or other systems languages for integrations and high-throughput infra.
Salary & Benefits
- Flexible work environment - ClickHouse is a globally distributed company and remote-friendly. We currently operate in over 20 countries.
- Healthcare - Employer contributions towards your healthcare.
- Equity in the company - Every new team member who joins our company receives stock options.
- Time off - Flexible time off in the US, generous entitlement in other countries.
- A $500 Home office setup if you're a remote employee.
- Global Gatherings – We believe in the power of in-person connection and offer opportunities to engage with colleagues at company-wide offsites.
Equal Opportunity Statement
ClickHouse provides equal employment opportunities to all employees and applicants and prohibits discrimination and harassment of any type based on factors such as race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
