Senior Big Data Engineer Location: SFO, CA (Remote is fine) but candidates need to work on PST Hours.
C2C Requirements and Direct Clients / Prime vendor / Implementation
Job Title:?Senior Big Data Engineer
Location: SFO, CA (Remote is fine) but candidates need to work on PST Hours.
Duration:?Long Term
Skill Sets – Python, SQL (expert level),?Spark and Scala (intermediate).
Experience: 10+ Years
Expertise:
10+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD
Experience with distributed processing technologies and frameworks, such as?Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3)
Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective product solutions
Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks
Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse)
Excellent written and verbal communication skills
?
A Typical Day:
Design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources, including user interactions, financial details, and external data feeds.
Develop data models that enable the efficient analysis and manipulation of data for merchandising optimization. Ensure data quality, consistency, and accuracy.
Build scalable data pipelines (SparkSQL & Scala) leveraging Airflow scheduler/executor framework
Collaborate with cross-functional teams, including Data Scientists, Product Managers, and Software Engineers, to define data requirements, and deliver data solutions that drive merchandising and sales improvements.
Contribute to the broader Data Engineering community at Airbnb to influence tooling and standards to improve culture and productivity
Improve code and data quality by leveraging and contributing to internal tools to automatically detect and mitigate issues.
Apply here : https://talenthubsolutions.com/
--
6 个月May i know who is the client and what is the best Rate for this position