How can you prioritize which game mechanics to prototype first?

Without a doubt, the core mechanics should always be the first thing you build. That way as you continue to develop and iterate, it gets polished with each iteration.

It is crucial to adopt an MVP (Minimum Viable Product) approach. Simplify the initial project to the maximum, allowing for incremental progression in development. For instance, when creating a roguelike game, what is essential? Combat? Enemies? Different levels? Inventory? Farming? Crafting? Eliminate what is not strictly necessary, focusing solely on the MVP, covering, in this case, combat, enemies, and distinct levels. Later on, it is possible to expand and incorporate other elements, such as inventory, farming, and crafting, as the project evolves.

Before you even go to the MVP stage, make a "walking skeleton". The difference is the target - the walking skeleton is for the internal team, devs and game designers, while MVP is something ready to show to investors/stakeholders/end users. Focus on creating the core mechanic that will be engaging enough by itself. Everything around it - muscles and skin, will support the skeleton but won't work on its own.

The core loop is exactly where players will spend most of their time. Your player controller and how users interact with your game are crucial. You should have a polished controller with basic features to test your game idea. The player controller should be enjoyable, even without elements in your game; it's the most important part. Many successful games like Cuphead, Jump King, and Celeste have unique controllers that make them special.

I always create a MoSCoW Analysis before any other pre-production materials so I can clearly outline what my goals and expectations are for whatever project I'm working on. It also helps me communicate my vision to the others in my team so we can work much more efficiently on our most important goals and continue from there. Efficiency is the key word there, documents like a MoSCoW Diagram, Gantt Charts and Design Docs not only define what the mechanics are but also how essential they are to building the final product, making it much easier to choose what mechanics are most important and have to be completed first.

MoSCoW is a simple technique yet it's not data-driven and does not balance value and tech effort well. WSJF is not a simple one, but it covers those two points and fits MVPs and new products as well. Putting WSJF simply, you evaluate the duration (or effort you put into a task) and the cost of delay and then divide one by another, getting the job with the highest value.

A risk register is a crucial piece of any software development, but often overlooked. Get the whole team involved in providing potential risks as you will be surprised at certain blind spots if this was left only to the devices of the producers/directors. By multiplying each risk's probability x impact you can get a risk score that will help you to prioritise which risks need monitoring the most when prototyping.

I'm glad this piece of advice is out there, because it has become invaluable to my practice. A misconception is that your prototypes need to be first steps towards a vertical slice. Prototypes are tools to answer questions. And it's often best to ask questions that matter first. As someone used to say to me, "bad news doesn't get better with old age." Figure out early what you can and can't do; what works and what doesn't; what's good and what needs work.

As much as new developers want to jump in and create their game, prototype iteration is very important! Maybe you figured out a certain mechanic worked and playtesting it soon uncovered that it didn't mesh well with certain colliders. During the prototype phase, you can then see if there is a fix without having to sacrifice days/weeks of development time.