This role is a technical, system design, architecture and leadership position which aims to address big data problems and requirements. The primary objective of the role is to help describe the structure and behaviour of a big data solution and how that big data solution can be delivered using technology options. Together with a team of software engineers and other experts, he needs to be hands-on with Hadoop applications (e.g. administration, configuration management, monitoring, debugging, and performance tuning) and evangelize devops.
It is a link between the needs of the organization and the scientists, analysts and engineers. He is responsible for managing the full life-cycle of a Hadoop solution and is one of the key stakeholders on information architecture. This includes creating the requirements analysis, the platform vetting, design of the technical architecture, design of the application design and development, testing, deployment but primarily lead operations of the proposed solution.
Ideally, a lot of experience gained in normal solutions architecture and/or systems engineering (~10-15) years of working experience is desired but not necessary. We understand that big data systems are still not common but becoming one quickly. A software engineer with adequate exposure to platform architecture, the right mindset and with exposure to but not limited with Hadoop (HDFS/Spark/Yarn), MapReduce, Hive, HBase, MongoDB, Cassandra, Kafka and high traffic web systems will be qualified. Experience in technologies like Impala, Oozie, Mahout, Flume, ZooKeeper and/or Sqoop is a plus.
Additional requirements include good programming foundation using 2 or more of the following programming languages • Java, Linux, PHP, Ruby, Phyton and/or R. Worked with ETL tools such as Informatica, Talend and/or Pentaho is a plus. He or she should also ideally have experience in designing solutions for multiple large data warehouses with a good understanding of cluster and parallel architecture as well as high-scale or distributed RDBMS and/or knowledge on NoSQL platforms.
Experience with AWS and related technologies will be necessary such as Elastic Map Reduce, RDS, RedShit, EC2 etc.
In summary, the overall qualification is for a highly skilled systems/devops engineer or architect with cross-industry, cross-functional and cross-domain know-how.
• Design big data solutions as a whole or in part
• Monitors and governs the implementation.
• Manage and monitor data security and privacy concerns that could arise and that should be taken care of from the start.
• Benchmark systems, analyse system bottlenecks and propose solutions to eliminate them;
• Clearly articulate pros and cons of various technologies and platforms;
• Document use cases, solutions and recommendations;
• Help program and project managers in the design, planning and governance of implementing projects of any kind;
• Perform detailed analysis of business problems and technical environments and use this in designing the solution;
• De-facto directly responsible individual for all backend systems
• Excellent written and verbal communication skills;
• Able to explain the work in plain language;
• Work creatively and analytically in a problem-solving environment;
• Self-starter;
• Team work, especially with highly diversified disciplines.
• Must be a proponent of DevOps practices
• Highly sensitive to system availability and scalability
• Understands that automation is not optional
• Understands that all tools and scripts must be in version control
• Triple checks operations, prior to execution in production.
• Understands configuration and change management
• Able to work in a fast-paced agile development environment.
• Consistent and has passion for understanding data
• BS or MS in computer science, math, physics and/or equivalent relevant skills/experience