ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)

Job title: ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)Location: London; full in-office working as defaultStart date: ASAPCompensation: Salary and Equity options available for discussionApplication deadline: 08/11/26 at 23:59 UK time___________________________________________________________________________We’re looking for an ML Systems Engineer to collaborate in training our Lumen models – our open‑source–based software engineering LLMs. This is a unique, and truly interdisciplinary role that involves developing and deploying our reinforcement learning (RL) training environments, working on large scale synthetic data pipelines and building our ML platform to enable training large models at scale in distributed environments. We want to make sure that the models we train are the best SWEs in the world - this doesn’t just mean training them to get the right answer, it means training them so that they write readable, maintainable code, that fits with the architectural patterns already present in the codebase. We believe we’re now in the anti-slop era of coding agents, where data, RL environments and opinionated reward functions will shape the future standards of SWE models. If this sounds exciting, then this could be the role for you.___________________________________________________________________________What you’ll doParticipate in end-to-end training of models:Supervised fine-tuning on curated code and conversation datasets.RL on top of those models to align them with software-engineering objectives.Develop and manage synthetic data generation pipelines to curate datasets that will underpin future training runs.Design, build and deploy containerized services using Docker and platforms like Kubernetes to enable our RL infrastructure.Build and maintain our training platform that enables us to conduct model architecture experiments at scale, train models, and run evaluations on them.Ideate on novel and opinionated reward functions for the training of SWE agents.You’ll collaborate closely with infra, product, and research to decide what to train next, how to train it, and how to measure whether it’s actually better for engineers.___________________________________________________________________________What we’re looking for (must-haves)Strong software engineering or computer science background:Typically 3-5 years of experience.You can read, debug, and write non-trivial production code (you’ll mainly be working across Python and Go).Experience with tools like Docker and container management/orchestration platforms, like KubernetesExperience with at least one major cloud-computing platform like GCP, AWS or AzureYou care about code quality, correctness, and maintainability as much as model metrics.Data engineering instincts:Comfortable working with large-scale datasets, object storage, dataset sharding, and filtering.Know that data quality and sampling strategies matter as much as architecture.Clear communication and ownership:Can take a vague modelling goal (“make Lumen better at X”) and turn it into a concrete plan of experiments.Comfortable documenting decisions and walking others through tradeoffs.___________________________________________________________________________Nice to have (bonus)You don’t need all of these, but the more you have, the more you’ll hit the ground running:Knowledge of PyTorch/Tensorflow/JAXExperience with synthetic data generation pipelinesExperience with data tooling like SQL, Apache Iceberg and duckDBExperience training LLMs in distributed environmentsSafety, robustness, and reward shaping:Experience with LLM-as-a-judge, reward hacking detection, or robustness evaluation.Open-source contributions or research:Contributions to open-source LLM tooling, RL libraries, etc.___________________________________________________________________________Why join CosineThis is a rare opportunity to work across the full model-training stack in a high-ownership environment, while still developing deep expertise in the area that interests you most. You’ll be close to the models and the people using them, with the freedom to move from an idea to a production system rather than stopping at an experiment, paper or benchmark. If you care as much about making research work reliably in the real world as you do about developing the underlying techniques, Cosine offers a place to do both.___________________________________________________________________________Cosine is an equal opportunity employer.We value diverse backgrounds, perspectives, and ways of thinking, and we’re committed to creating an inclusive and respectful workplace.We encourage applications from anyone who meets the role requirements, even if you don’t meet every single qualification. If you need reasonable adjustments at any stage of the hiring process, we’re happy to discuss them.___________________________________________________________________________Compensation, Benefits & Ways of WorkingWe’re an in-office team, five days a week, by design. We believe the work we’re doing benefits from being together, collaborating closely, and building shared context.What you can expect:Competitive salary, benchmarked to the marketEquity / share options, so you share in the upside you help create30 days’ holiday + bank holidaysWork hard in the office, collaborate closely, and switch off properlyDog-friendly office - bring your dog to workWeekly team breakfast & lunchMonthly socialsPensionHigh-quality equipment to do your best workWe care about focus, sustainability, and doing great work — not performative overwork. We value people who show up, contribute thoughtfully, collaborate well with their colleagues, and then go home.This role won’t suit everyone. But if you’re looking for a fast-moving environment, close collaboration, and a team that takes both the work and work-life balance seriously, it’s a great place to be.___________________________________________________________________________Agency & Data Protection NoticeTo comply with UK GDPR and our internal data-protection and equal-opportunity obligations, we only accept candidate applications and agency submissions via our Applicant Tracking System (ATS). This ensures appropriate privacy notices, lawful processing, audibility, and consistent retention controls.Any CVs or candidate details received outside the ATS (including via email, Slack, or direct message) will be treated as unsolicited, will not be considered as part of the recruitment process, and will not give rise to any fee or payment obligation.