Ranga's Spark Project Template Generator

Hi All,

I have created open source spark project template generator application. By using this application you can based on templates you can create N number of applications with in seconds or minutes.

Advantages of using this application:

1. Create?sbt?or?maven?or both build based Spark applications.

2. Through single configuration (config.yaml?or?config_all_apps.yaml) you can create?N number of Spark Applications.

3. Supports various Spark templates like hive, hbase, kudu, various file formats etc and you can create your own templates.

4. Generates both?Scala?and?Java?based code as of now.

5. Generate the?run script?to run the spark application.

6. Deployment steps are mentioned in?README.md?file.

7. Built in?Scala Test?code for Default template.

Source code:

You can find the source code below github url.

Github Source Code

Sample Applications:

I have created bunch of spark applications using this application.

Spark Applications

Please let me know is there any feedback on this application.

You can contact me using following linkedin for any clarifications and doubts.




saikiran Durga

Engineering Lead at Fedex

3 年

Thanks Ranga for sharing, this helps a lot

回复

要查看或添加评论,请登录

Ranga Reddy的更多文章

  • Apache Iceberg History & Spark Supportability Matrix

    Apache Iceberg History & Spark Supportability Matrix

    1. Introduction The Spark and Iceberg Supportability Matrix provides comprehensive information regarding the…

    2 条评论
  • Apache Spark Supportability Matrix

    Apache Spark Supportability Matrix

    1. Introduction: One of the most common challenges faced while developing Spark applications is determining the…

  • Spark History Server Docker Image

    Spark History Server Docker Image

    A Sample Docker image for Spark History Server to deploy and manage the Spark Event Logs locally. Step1: Pull the…

  • Shell Script to generate Random CSV data

    Shell Script to generate Random CSV data

    Source Code: https://gist.github.

    3 条评论
  • Spark Configuration Generator

    Spark Configuration Generator

    Hello Spark Enthusiast Are you looking for generating the Spark Configuration based on Resources (Hardware…

    3 条评论
  • Create your first Airflow DAG

    Create your first Airflow DAG

    Let's start creating a Hello World workflow, which does nothing other than sending "Hello World!" to the log. A DAG…

    22 条评论
  • Install Apache Airflow on Mac OS

    Install Apache Airflow on Mac OS

    Airflow is written in python, so python needs to be installed in the environment, and python must be greater than 2.7…

    17 条评论
  • Spark code to create a random sample data

    Spark code to create a random sample data

    In this article you will learn how to create a random sample data by using spark. import org.

社区洞察

其他会员也浏览了