Leading, and Pairing on, ML projects

Most machine learning projects sit squarely in the intersection between "spend two years on this and get back to me" research and overly pragmatic "let’s ship it tomorrow" engineering. Each company has a preference for where on the spectrum they are; some dedicate full research teams to optimizing the last .001%, while others use whenever open-source code they can find to classify MNIST digits and call it a day. For me, I like to be on, and lead, teams that sit in the middle as full-stack machine learning engineers. However, the difficulty with the middle is that the work required is often very wide encompassing pulling data, exploring data, modeling, building infrastructure, serving, retraining …. The list goes on. Given this, what is the best way to staff people on these types of projects? I’ve tried lots of approaches; Kanban, waterfall, research-focused, variants of Agile. They can all work.

tl;dr Staffing projects all depends on your team dynamics and the individuals that are working with you

However, I have found expectation setting through project leads (not a new concept) and project pairs (a newer concept?) to be helpful for this type of work.

Above are all the different phases of a project. As you can see, it covers a wide variety of areas.

Project leads are formally the individuals responsible for ensuring a project is seen to completion, communicating the status of a project including any changes to the project timeline, as well as leveraging data to assess the impact. These people are your “I trust you to complete this without hand-holding the entire time.” In addition, project leadership is a great opportunity for junior developers to work on smaller features. Here are some expectations I’ve found useful.

Lead Core Expectations

For all projects

• Providing in-depth, opinionated engineering knowledge:

1. You use the right media - code snippets, architecture diagrams, whiteboard - to communicate ideas to audiences of varying technical backgrounds.
2. You provide API documentation

• Working through the requirements for a new project:

1. You are responsible for gathering the requirements for the system including model performance requirements, latency expectations, architecture technologies

• Ensuring your project pair has context on the work and the plan for sequencing work:

1. Understanding which work can be parallelized (model training, hyperparameter tuning)
2. Can you provide the appropriate context for the individuals you are working with

• Remaining two steps ahead of anticipated questions. Leverages the correct audience, content, and tone while communicating

1. Email, Slack, Zoom, in-person, PowerPoint, excel you choose the communication updates appropriate for the audience

• Shipping the project

1. You can be relied on for getting projects across the finish line and leveraging others to do so

• Supporting rollout

1. Meet with stakeholders to discuss a rollout strategy and timeline
2. Be the primary point of contact for any bugs or quick updates needed during the rollout
3. Ensure we have a testing plan in place
4. Ensure we have success metrics defined

Rolling up their sleeves next to the lead, project pairs are the second brain. With the work spanning many different skillsets, tech stacks, and stakeholders, I find it useful to explicitly assign pairs on projects. Both for personal development, “hey, this technology looks cool,” removal of the bus factor, “only Peter knows how our Resnet backbone was trained,” and PR reviews, “There are 15 PRs that have been open for more than a week.” The expectations of a pair depend on the stage of a project.? Some parts of the machine learning lifecycle are well suited for pairing such as model experimentation, while others, deployment, and infrastructure, require the pair to be the primary reviewer on design documents to offer critical feedback. Generally, I’ve found that pairs spend 10-15% of the total time on the project assisting and gathering context.

Pair Core Expectations

For all projects

• Responsible for ensuring code is shipped in a timely manner to the production

1. PR reviews, design reviews?

• Has relevant context for the project

1. Attend meetings with stakeholders
2. Weekly syncs with the project lead
3. Help write acceptance criteria for upcoming stories
4. An understanding of the proposed model algorithm including relevant tradeoffs
5. An understanding of the core data required (if applicable)
6. An understanding of the architecture choices made and why
7. Able to ship and deploy code if needed

• Helps works through project sequencing

1. Parallelize which parts?
2. What comes first?
3. Who needs to approve?

Pair Secondary Expectations

Project dependent

• Parallelizing model training with the project lead. Two brains working to find the best model > one brain.
• Writing project-specific code

1. Examples include: data transform pipelines, infrastructure setup, integration review, micro/macro tests

• Writing architecture diagrams or other documents outlining decisions made during the project

1. You and the project lead can discuss who should handle this

Anticipated Q&A

Q: Shouldn’t everyone know how everything works?

• I think we can all agree that’d be wonderful. However, given limited time and energy, and the fact that each project is very wide, it’s useful to have someone tagging along all of the steps instead of someone popping in here and there without full context

Q: What about the rest of the team, should they have a say?

• Of course, but not for all decisions. At the highest level, the team, depending on size, should be able to articulate the business problem/value for everything everyone is working on. You can accomplish this through other venues such as sharing design documents team-wide or a recurring showcase meeting. However, you also need other engineers to work on their own projects.

Q: You’re making a lot of assumptions about the communication preferences and working styles of people.

• Definitely. See the tl;dr above. With that said, I do expect people I work with to be able to work with others in either a lead or pair capacity.

Thanks for reading!

?