The Good, Bad, and Ugly of a Pilot Test

**Originally published here**

ExEC is an experiment to see if we can revolutionize how entrepreneurship is taught in college classrooms.

Halfway through our first pilot semester, we wanted to share the results so far – warts and all – so the entrepreneurship education community at large can learn along with us.

The Numbers

We’re grateful to have 10 schools across the United States and Canada piloting with us this Fall:

• Rowan University
• Brandon University
• East Carolina University
• Southern Alberta Institute of Technology (SAIT)
• University of South Alabama
• Gulf Coast State College
• Xavier University
• John Carroll University
• Susquehanna University
• Georgia State University

Across these institutions, we have nearly 500 undergraduate students using the curriculum.

Just as ExEC challenges those 500 students to do, we’re testing the curriculum’s assumptions and iterating its design. Here’s what we’ve found so far.

The Good

Our pilot professors and students have told us ExEC stands out with respect to other curricula in a couple areas:

Developing the Entrepreneurial Mindset

We want students engaging with their customers, discovering problems that can be solved instead of just thinking of new ideas to create.

One professor told us:

“More than anything, I’ve enjoyed that we have spent 4-5 weeks exploring the issue of problem solving. In previous classes, students have been convinced they had the right solution to a problem by week 2 and no matter what research they found, they wouldn’t pivot appropriately given the new evidence.”

Students are also sharing their excitement at better understanding how entrepreneurs think. One student said:

“This activity made me look at the creation of a product in a different aspect than I have before. It allowed me to think of solving a problem and not just creating a product to create one. It needs to be something that people will actually use. It made it easier for me to be creative and think more like an entrepreneur.”

Another student told us:

“The exercise was a very clear, somewhat concise explanation of the mindset needed for successful entrepreneurship. It shows the clear relationship between successful startups and outlines the key consistencies for success.”

And perhaps our favorite student feedback:

“[The  Business Plans vs. Business Experiments exercise] made me think like a kindergartener again and that made me excited.”

Replacing Lectures with Experiences

ExEC provides professors with comprehensive lesson plans, and constant support, so you can create experiences like this that will excite students about experiencing entrepreneurship.

One of the exercises uses marshmallows to teach students about the danger of hidden assumptions, and why business plans lead to failure more often than not:

Pilot students have been sharing their excitement with the exercises as well. They are seeing how they can apply what they are experiencing in class:

“This really showed me what it takes to develop an idea. Also helped me get more in touch with who I’m trying to target with my idea and how it can help [him/her].”

“I can use the [exercise] for all business ideas that come to mind and when analyzing other companies.”

Students have shared how the exercises have shifted their thinking:

“I feel I will perform better in the future when completing a project because now I know the value of prototyping, also, now I understand the importance of identifying the hidden assumptions which cause many times good ideas to fail.”

“Knowing how feelings play a part in buying and decision making is interesting and will help with my business model. I did not really think or relate the two before this class.”

“This provides a clear understanding of what it takes to solve a problem and come up with ideas to solve those problems. It showed me some key resources that I did not know before to help start a company through problem solving.”

Getting Students Interviewing Customers

We want students having real conversations with real customers about the customer’s problems. This is the essence of entrepreneurship, and a skill we heard most professors struggle teaching.

One professor told us:

“Given the previous exercises on identifying the early adopters and clarifying the problem statement, [The  How To Ask for Interviews exercise] was a very positive exercise. Students were able to quickly identify the interview channels that their early adopters might use (social media, blogs, interviews) and plan how to initiate that conversation using the strategy outlined in this exercise. By the end of today, the students felt very confident about getting out and learning about the problem.” 

Our interviewing exercises push students to think about learning from actual customers, instead of industry or product “experts”, as shared by one professor:

“I LOVE [ Who Are Early Adopters?] exercise!!!  So many times, I have had conversations with my students who are going to interview their dad, friend, someone who works in the industry and this exercise really takes the time to dispel this myth that these are important.”

The students are also realizing how powerful interviews are, especially in comparison to surveys, thanks to an ExEC exercise that makes them survey and interview customers, and compare the results:

“The [ Student Challenges Survey exercise] is showing how surveys do not capture the full picture from a consumer whereas an interview lets the customer give more feedback.”

After half a semester, we are confident the pilot students are engaged in their class experience. Through that engagement, we see them developing an entrepreneurial mindset, and honing their customer interviewing skills.

The Bad

We preach iteration because there’s no way to get everything right the first time around – and that’s the case with ExEC. Here are a couple areas we need to focus on going forward.

Less is More

We created too much content. We originally wanted to arm our professors with more experiences and exercises than they could ever use, so they could build a customized syllabus specifically for their class.

That strategy has started to backfire as some professors have, understandably, began feeling pressured to cover a lot of material in a limited amount of time. One told us:

“I was extremely nervous . . . to teach a class that had so many new components that I was learning day by day.”

Another shared:

“I think there are too many lessons on interviewing, although I see its utility.”

ExEC has 30+ experiential exercises, which is simply too many for one semester, especially when life readjusts the class schedule, as it did with Hurricane Irma for a couple professors.

The Fix

Based on the exercises students find most impactful, we’re streamlining ExEC’s content to focus on a subset of activities.

Restricting Access

We were so excited to share ExEC with as many schools as possible, we initially weren’t as rigorous as we should have been in restricting access.

ExEC has a lot of moving parts, several of them untested before this semester. We should have started with a slightly smaller, more targeted, pilot cohort so we could iron out ExEC’s wrinkles (details below) more efficiently.

The Fix

Having been through trial by fire this semester, we think we have a handle on the major issues. That said, we’ll be limiting access to our upcoming Spring Pilot, just to err on the side of caution.

The Ugly

There is one area we really missed the mark this semester. Because of it, we’ve already started the redesign process.

Poor Design Choices

We made some incorrect assumptions about the technical comfort of some of our students and professors – which really means we made poor design choices on our end.

Our pilot professors and their students are less familiar with technology than we anticipated. We expected more feedback like this from one professor:

“I have taken one class at a time, one new element at a time and really enjoyed exploring the new materials.”

But one professor told us:

“I like to think I’m not stupid, but working through this to get it ready for my students makes me question that.”

And one student told us:

“I’m absolutely thrilled that [I] bought a program coded by a team of incompetents.”

While our professors certainly aren’t stupid, and we (hope we) aren’t incompetent, any experience that makes even a subset of customers feel that way needs to fixed immediately.

The Fix

We’ve already begun making to several exercises, but there’s more work to do.

We’ve started redesigning both the professor and student experiences from the ground-up; while the content will largely remain the same going forward, the way professors and students interact with it will be completely revamped.

Takeaways

So far, this has been a perfect pilot!

Not perfect in that we got everything right – we certainly haven’t – perfect in the sense that this is what pilots are for. With the help of our amazing pilot professors, we’ve been living the Build, Measure, Learn loop.

We’re really excited about ExEC’s and while we haven’t gotten everything right so far, we’ve made some great progress on some of the hardest parts:

1. Developing the entrepreneurial mindset
2. Replacing lectures with experiences
3. Getting students interviewing customers

Next up, we’ll polish our rough design edges so everyone feels confident engaging with the content!

Want to Shape Entrepreneurship?

As mentioned above, we’ll be limiting access to the Spring Pilot, but if you’re a progressive entrepreneurship professor willing to get your hands dirty in the name of improving entrepreneurship education…

Check out  ExEC  and schedule a preview.

We’ll accept a handful of programs into the Spring Pilot, which will not only get you early access, as you can see, you’ll also play a significant role in reshaping how entrepreneurship is taught at colleges around the world.