Can you really build features in Apache Spark in 1/3 of the time?
Can you really build features in Apache Spark in 1/3 of the time you currently can?
Often when there is a new technology in software, people get excited and lots of figures are banded around stating how much it saves developers, or how much more efficient it makes them.
The challenge is there are so few scientifically controlled experiments to actually prove this. That is why sometimes new tools or utilities are taken up, without anyone actually confirming they will make things faster. Often new skills are required to be learned, harder to recruit for and sometimes new technologies turn out great for building hello world type features, but actually try something complex and it suddenly becomes clear the tool/software needs maturing.
Well here at www.claritize.com, we have decided to share what we have found using Apache Spark. We recently migrated our old platform to Apache Spark, and we had to develop exactly the same feature, with the same requirements in both platforms. Our old platform we know well, and we now know Apache Spark just as well. So - this new feature was developed by a team of one developer, in Apache Spark, and a team of 3 in our old Platform. They both developed the feature independently and at the same time, in different locations, the experience level of the developers is comparable (actually in the team of 3 - we had one super experienced developer). So everything as equal as possible.
The results? In Apache Spark we built the feature with one developer taking less than 2 weeks. It took 3 developers 6 weeks to do exactly the same in our old platform.
So - the conclusion? Comparing Apache Spark with our old platform, we built a feature in 1/3 of the time with 1/3 of the resources. If this scales up - a 12 month feature could potentially be built in 4 months. Our own experience with Apache Spark tells us this feels about right. But - and here is the but, using our old platform - meant we weren’t comparing Apache Spark vs no platform, so we already had utility functions and code, so Apache Spark is probably an even faster leg up that our statistics tell us.
If you wish to know more, we are more than happy to share the detail, so please do contact us, and in return, we would love to know much how faster you have found it to develop in Apache Spark.
Regards,
Andrew
CEO Claritize