Senior Chaos Engineer - Remote

Overview

At Goodnotes, we believe that every individual holds untapped potential waiting to be unleashed. By reimagining the way we interact with information, we’re merging human creativity with the breakthrough capabilities of AI. Our renewed vision and mission drive us to create the best medium for human and AI collaboration, empowering users to explore new dimensions of productivity, creativity, and learning. Join us on this journey as we transform digital note-taking into an inspiring and innovative experience.

In Short

• Define the chaos engineering strategy at Goodnotes, including tools, safety practices, and long-term roadmap.
• Design and run fault injection experiments across mobile and backend systems, targeting failure points in user flows, APIs, and infrastructure components to surface hidden risks.
• Simulate real-world issues like latency spikes, dependency outages, cascading failures, and resource exhaustion.
• Build and scale tooling for automating experiments, tracking outcomes, and improving observability.
• Establish clear guardrails and blast radius controls to ensure experiments are safe, measured, and reversible.
• Collaborate across engineering teams to identify critical flows, formulate hypotheses, and stress-test assumptions.
• Facilitate resilience drills and chaos game days, driving cross-team engagement and response readiness.
• Document findings, communicate insights, translate chaos learnings into actionable improvements, and influence our engineering teams to enact recommended changes.
• Help shape the future of the chaos engineering function — including mentoring and hiring as the team grows.

Requirements

• Proven experience with chaos engineering or fault injection, ideally in distributed, production-scale environments.
• Comfortable with iOS platforms, mobile networking, and understanding how client-side failures impact backend systems. Strong experience with Swift programming.
• Strong understanding of resilience patterns (e.g., circuit breakers, bulkheads, timeouts, retries) and system failure modes.
• Prior involvement in incident postmortems, war games, or reliability reviews.
• Comfortable building tools or scripts to automate chaos experiments and analyze system behavior under stress.
• With your scientific mindset, you love forming hypotheses, testing limits, and uncovering how systems really behave at the edge.
• You’re excited to build a program from scratch, not just join one.

Benefits

• Remote friendly
• Flexible working hours and location
• Medical insurance for you and your dependents
• Great annual leave allowance
• Meaningful equity in a profitable tech-startup
• Budget for things like noise cancelling headphones, setting up your home office, personal development, professional training, and health & wellness
• Sponsored visits to our Hong Kong or London office every 2 years
• Company wide annual offsite
• Fantastic maternity/paternity packages and allowances