The critical role of beta testing: the Penpot Plugin System case

Communities play a key role in shaping tools and services.? In the world of open-source products, the community acts as both creator and contributor.

There are many ways to contribute, and feedback is one of them. The more diverse the feedback, the more complete the tool becomes. At Penpot, we invite everyone to get involved based on their individual backgrounds or needs. We believe that every contribution can make a significant difference in improving the platform.

This article highlights a specific type of contribution: beta testing. This stage is vital, as end-users help identify weaknesses and reduce uncertainties before the product is officially launched.?

We recently released the Penpot Plugin System, an improvement that allows creators to build and share custom extensions. This milestone was reached thanks to the tremendous effort of the team and the significant contributions of the beta testers, who dedicated their time and energy.

Why does beta testing matter?

This stage is crucial, especially in a product like ours. As our product owner Andrés González says, "we're not just building a product, but also a community". This means gathering feedback and creative input from the community to help create the product we are all going to enjoy.?

"We're not just building a product, but also a community".

Plugins, in particular, give the community greater control and opportunities to enhance Penpot. Understanding end-user feedback and their needs within the Plugin System was key to the successful launch of this new feature.

What kind of rewards can beta testers expect?

While beta testing often includes rewards like discounts or free products, in this case the reward was early access to a new feature and the chance to contribute to improving the product they would personally use. At Penpot, beta testers help shape the tool they’ll enjoy for free.

What other benefits does the community receive? Contributions can be motivated by different reasons. Clara García Vi?ola , product designer at Penpot, highlights some in this Why Open Source?? post. These include skill development, fostering innovation, learning and teaching, and building a reputation.

How to run a successful beta test?

After our internal (alpha) testing, where we demonstrated the plugin system potential, we were excited to move to beta testing, where volunteers tried the product in real-world settings.?

Here’s how we conducted our beta testing:

1. Set our goals

We clearly outlined what we expected from contributors, specifically to create their own plugins: bug identification, feature feedback, performance assessment, etc.

2. Assemble materials We created complete documentation for our beta testers, including the goals and expectations, a "How to do the beta test" guide, potential challenges, and contact details. For a successful collaboration, it’s essential that volunteers clearly understand their responsibilities and what lies beyond their scope.

3. Call for creators We reached out to creators inviting them to test the new Plugins System and asking them to share their honest feedback. Open sharing is key to addressing real needs in an authentic way. What matters most is inviting the right group that aligns with our ideal audience.?

4. Gathering feedback As they completed their testing, we gathered everyone's feedback. Although we had tested the Plugins System with our own plugins, receiving our first community-created plugins was especially exciting.

Some of them were:

• All Icons by Belar: A vast library of icons.
• QR Code by Thierryc: Generate QR codes directly in Penpot.
• Day & Night by Christoph: Set up color palettes for light and dark modes.
• PDF Viewer by Girafic.

5. Analyzing and taking action

After several months of careful follow-up, with nearly 100 beta testers, we addressed 50 smaller issues and turned their feedback into meaningful improvements, such as better search functionality, simplified installation, and clearer documentation for plugin creation.?

6. Wrap-up and release!

Once we shared the updates with the community, recognizing the high value of their contributions, everything was in place for the final launch of the Penpot Plugin System.

Bonus: a plugin contest to keep learning

Alongside the launch, we hosted a plugins contest running until December 15, 2024. This action will allow us to:

1. Introduce the plugin system and encourage everyone to give it a try.
2. Refine the tool and the bulk-upload process as plugins are submitted for the contest.

?

This is the recipe we follow, and well, we continue to adjust it as needed.

What other tips would you recommend for conducting successful beta tests? Have you ever participated in a beta test for products??

Ps. This article kicks off The Penpot Brief. Stay tuned for more in the coming weeks!