Big Data Admin

Job Summary

As a Level 3 Big Data Services professional, you will lead technical tasks, drive collaboration, and support service improvements to maintain reliable system infrastructure aligned with SLAs. You will engage in issue resolution, technology adoption, and team development.

Responsibilities

• Lead and delegate technical tasks within the team to resolve complex Big Data issues effectively
• Foster team collaboration and share best practices to enhance service quality
• Support and integrate new technologies to ensure consistent service delivery across streams
• Propose and implement service improvements for Big Data services organization-wide
• Document, review, and maintain technical information to support team knowledge sharing
• Provide proactive and reactive technical support to maintain system availability and reliability per SLA
• Actively engage and drive resolution during high severity incidents
• Review technology changes to identify and mitigate potential risks
• Support team members in diagnosing technical issues within your expertise
• Understand global IT structure to anticipate organizational interdependencies
• Collaborate with technical peers, development teams, service managers, architects, and project teams on technology roadmaps and projects
• Facilitate transformation projects and recommend future improvements and changes
• Ensure production readiness and operational approval for new projects and solutions
• Lead technical response for complex or high severity incidents within your role scope
• Research and evaluate new technologies and trends for potential integration into operations
• Assist in identifying technically skilled candidates for open roles
• Coach and develop partner resources to improve quality and productivity
• Build and maintain Hadoop stack infrastructure, including installation of updates, patches, and version upgrades
• Manage cluster deployments, including node creation and removal
• Perform HDFS backups and restores to ensure data integrity
• Manage Hadoop user access, including Linux user setup, Kerberos configuration, and permissions for Hadoop components
• Support and maintain HDFS and monitor filesystem health
• Manage and review Hadoop log files for troubleshooting and performance insights
• Monitor Hadoop cluster connectivity and security to maintain system integrity
• Administer and operate Kafka services within the cluster environment
• Tune cluster and MapReduce routines for optimal performance and resource utilization
• Configure and screen cluster jobs to ensure efficient processing
• Conduct capacity planning to meet current and future workload demands
• Utilize ElasticSearch Stack for data indexing and search capabilities
• Collaborate across teams to guarantee high data quality and availability
• Serve as liaison between developers and build/architecture teams to facilitate smooth operations
• Document best practices and maintain a comprehensive knowledge base for continuous team learning

Required competencies and certifications

• Minimum 8 years in system administration performing system monitoring, storage capacity management, performance tuning, and infrastructure development
• Minimum 1-2 years deploying and administering large Hadoop clusters
• Experience in the financial and banking industry
• Bachelor’s or engineering degree
• Working knowledge of Hadoop ecosystem components (Hadoop, Hive, Pig, Oozie, HBase, Flume, Sqoop) using automated and manual tools
• Ability to isolate and troubleshoot Hadoop service issues using system and Hadoop logs and monitoring systems
• Expertise in writing and debugging shell scripts, Perl, and Python scripts
• Ability to work independently and collaboratively on moderate to highly complex issues
• High technical aptitude with continuous skills improvement
• Ability to automate software/application installations and configurations on Linux servers
• Network and system administration experience, preferably in large-scale data center environments
• Excellent communication, interpersonal, and logical reasoning skills
• Customer service orientation and strong team collaboration skills
• Ability to work under pressure and commitment to issue resolution

Preferred competencies and qualifications

• French language proficiency
• Development experience in Java, Python, Perl, PHP, CSS, or HTML
• Experience with Spark development
• Experience administering Hadoop 2.0 clusters including YARN and YARN-based applications such as Spark
• Linux administrative skills including PAM authentication, disk quota, ulimit configuration, and security patch management
• Experience coordinating rolling OS-level changes with cluster administration tools
• Experience configuring and using cluster-wide monitoring tools for diagnosing issues and proposing enhancements
• Experience with Apache Ambari for Hadoop cluster management