Data Engineer @ OTA Insight in Ghent Metropolitan Area

What you will do:

We are looking for a Data Engineer who is passionate about complex data and its quality. Lighthouse relies on the Data Platform within its Engineering Organization for scalable, reliable, and secure data management. 

Where you will have impact:

• Design and develop scalable, reliable data pipelines using the Google Cloud stack.
• Ingest, process, and store structured and unstructured data from various sources into our data-lakes and data warehouses.
• Optimise data pipelines for performance and scalability.
• Implement and maintain data governance frameworks, ensuring data accuracy, consistency, and compliance.
• Create and maintain architecture RFCs and documentation
• Monitor and troubleshoot data pipeline issues, implementing proactive measures for reliability and performance.
• Collaborate with the DevOps team to automate deployments, with the data science team to productionalise models and fine-tune our airflow cluster, and with the analytics engineering team to decide on the data to be ingested into our data warehouse. As a platform team, we are expected to communicate effectively with teams across the entire engineering organisation.
• Mentor and provide technical guidance to other engineers working with data.

About our team:

Lighthouse is a data-driven company. Almost all business decisions here are backed by data. The Data Platform lies in the heart of the Engineering Organization of Lighthouse serving a big part of the data needs for the company. We manage state-of-the-art infrastructures, services and products and are constantly innovating to support all our business needs and products at scale.

Lighthouse’s Data Platform team has the following responsibilities and goals:

1. Scalability, reliability, and security: A good data platform should be able to handle increasing volumes of data and user demands, provide reliable access to data, and protect the data from unauthorised access or loss.
2. Enabling data-driven decisions and innovation: The ultimate goal of our team is to support the organisation’s strategic objectives, whether that’s making better business decisions based on data, developing new data-driven products or services, or improving operational efficiency.
3. User empowerment: A data platform should not just be a tool for data experts, but for all users in the organisation. This includes providing user-friendly tools, clear documentation, and training to help users access and understand the data they need.
4. Data governance and compliance: With the increasing importance and sensitivity of data, it’s crucial to have clear policies and procedures for managing data, as well as ensuring compliance with data protection regulations.

Technologies you’ll work with:

GCP, Python, BigQuery, Kubernetes, Airflow, dbt, dataflow, Kafka

 (feel free to ask us why these are the best fit for us)

What’s in it for you:

• A flexible working environment where you can work from home or, if you live nearby, at one of our seven offices around the world
• Unlimited time off: we believe in providing our employees with the flexibility and autonomy to manage their work-life balance effectively
• A motivation to deliver your best work as we have built a high-bar and very talented team of individuals that are friendly, creative, open-minded, and passionate about what they do
• Workshops and frameworks that help employees realise their full career potential
• A mandate to operationalize and democratise research through tools, training, and processes
• The opportunity to shape the products that more than 85,000 users rely on worldwide
• A choose-your-own mobility budget when you’re based in Belgium or a company car
• Extensive health insurance packages for you and your dependents prioritising our employees’ holistic well-being and healthcare needs
• Private pension funding through group insurance
• A compensation that values your work and which we will proactively keep competitive
• A referral bonus scheme when you bring new talent to our #bestteamever!

Who you are:

• 2+ years of professional experience using Python, Java, or Scala for data processing (Python preferred)
• Demonstrably deep understanding of SQL and analytical data warehouses (BigQuery preferred)
• Hands-on experience implementing ETL (or ELT) best practices at scale.
• Hands-on experience with data pipeline tools (Airflow, Luigi, Azkaban, dbt)
• Desire to continually keep up with advancements in data engineering practices
• Constantly improve product quality, security, and performance
• Catch bugs and style issues in code reviews
• Ship medium features independently

We welcome:

• Familiarity with GCP, Kubernetes (GKE preferred),  CI/CD tools (Gitlab CI preferred), familiarity with the concept of Lambda Architecture.
• Experience with Apache Beam or Apache Spark for distributed data processing or event sourcing technologies like Kafka.
• Familiarity with monitoring tools like Grafana & Prometheus.