At JetBrains, code is our passion. Ever since we started back in 2000, we have been striving to make the world’s most robust and effective developer tools. By automating routine checks and corrections, our tools speed up production, freeing developers to grow, discover, and create.
We are working on an ambitious new platform that provides AI capabilities to all JetBrains products. Our platform is based on in-house developed models for writing and coding assistance, as well as integration with models from our strategic partners.
We are looking for a skilled machine learning (ML) engineer who will focus on rigorous end-to-end AI system evaluation, including offline experiments, A/B tests, and leaderboards for our code completion, text generation, and information retrieval models.
We value engineers who:
- Plan their work and make decisions independently, consulting with others if needed.
- Follow the latest advances in ML, think long-term, and take ownership of their scope of work.
- Prefer simplicity, opting for sound, robust, and efficient solutions.
In this role, you will:
- Design and develop rigorous offline and online evaluation benchmarks that we can trust.
- Deploy end-to-end evaluation pipelines for in-house and external ML models.
- Be responsible for model selection and comparison with state-of-the-art methods.
- Analyze evaluation results and propose improvements to data and models.
- Communicate your findings and best practices to the entire organization.
We’ll be happy to have you on our team if you have:
- Expertise in the evaluation of generative AI methods.
- A good understanding of statistics and data analysis.
- Practical Python language skills.
- Familiarity with frameworks such as NumPy, SciPy, pandas, and Hugging Face Evaluate.
- Attention to detail in everything youdo.
- Great communication skill.
We’d be especially thrilled if you have experience with:
- Recent research on generative AI methods and their evaluation (especially for code generation and completion methods).
- ML frameworks, such as PyTorch, spaCy, and Transformers.
- Data annotation management, including crowdsourcing and in-house labeling.
- CI, workflow automation, and experiment tracking systems.
- The Kotlin programming language.
To develop JetBrains AI, we use:
- Git for source code management.
- AWS and GCP for infrastructure.
- Kubeflow for workflow automation.
- Continuous integration with TeamCity.
- Weights & Biases for experiment tracking and reports.
