Senior Big Data Infrastructure Engineer
Location : Dallas, TX
Do you want to tackle the biggest questions in finance with near infinite compute power at your fingertips?
G-Research is a leading quantitative research and technology firm, with offices in London and Dallas. We are proud to employ some of the best people in their field and to nurture their talent in a dynamic, flexible and highly stimulating culture where world-beating ideas are cultivated and rewarded.
This is a hybrid role based in our new Dallas infrastructure hub where we work on the latest technologies in a cutting-edge environment.
G-Research is seeking a DevOps focused Infrastructure Engineer, with deep experience in HDFS, YARN and the Hadoop ecosystem, to join its growing Big Data Platform Engineering team.
Big Data Platform Engineering manages a significant distributed Cloudera HDFS storage footprint, has recently adopted Airflow and has also started to explore other Open Source offerings, such as Ozone. In addition, there is also an overlap in technology support within the team, in products such as Trino, Spark, Grafana, Hive, Nifi and HBase to name a few.
As a Senior Big Data Infrastructure Engineer, you will help us deliver automation, feature enhancements and best practice security on our existing hybrid cloud platforms, as well as explore new technologies and integration with other Open Source products within our ecosystem.
There is a significant focus on configuration management, container orchestration, Infrastructure as Code (IaC) and CI/CD. In return, you will gain exposure to the latest hardware and software technologies in a forward-thinking company, which values innovation, personal development and training.
Key responsibilities of the role include:
- Developing, deploying and maintaining big data solutions within the Hadoop ecosystem, such as HDFS, YARN, Ranger and Hive.
- Delivering engineering solutions ensuring they are scalable, stable, performant, and operationally efficient
- Providing metrics, documentation, and self-service infrastructure to help our customers work at pace and get the most out of the platforms
- Proactively identifying and addressing system performance bottlenecks, security vulnerabilities, and areas for improvement to ensure systems are stable and robust
- Deploying and maintaining Airflow
- Participating in the out of hours rota, providing support and maintenance for our critical systems across the BigData ecosystem
Who are we looking for?
Whatever your technical background, good coding ability is essential for this role.
The ideal candidate will have the following skills and experience:
- Strong HDFS, YARN and Hadoop ecosystem knowledge, preferably on the Cloudera Distribution but also OSS Distributions
- Excellent coding skills, preferably Python
- Experience working with CI\CD pipelines, such as Jenkins and ArgoCD
- Experience working with Configuration Management tools, such as Ansible and Terraform
- Strong Linux, Hardware and Networking knowledge
- Experience deploying and running applications on Docker and Kubernetes, including the creation of Helm charts
- Experience working with distributed systems
Highly desirable but not essential:
- Experience working with Ozone
- Experience working with other Big Data technologies, such as Spark, Trino, Hive, Airflow, Ranger, Oozie etc
- Data Engineering experience (Spark application development, SQL)
- Team Lead experience
Why should you apply?
Market-leading compensation plus annual discretionary bonus
Informal dress code and excellent work/life balance
Excellent paid time off allowance of 25 days
Sick days, military leave, and family and medical leave
Generous 401(k) plan
16-weeks' fully paid parental leave
Medical and Prescription, Dental, and Vision insurance
Life and Accidental Death & Dismemberment (AD&D) insurance
Employee Assistance and Wellness programs
Generous relocation allowance and support
Great selection of office snacks, and hot and cold drinks
On-site gym and car parking
G-Research is committed to cultivating and preserving an inclusive work environment. We are an ideas-driven business and we place great value on diversity of experience and opinions.
We want to ensure that applicants receive a recruitment experience that enables them to perform at their best. If you have a disability or special need that requires accommodation please let us know in the relevant section.Apply