3rd Nordic WiDS Conference

Instrumentation is an often overlooked first step in the data life cycle. It’s the stage that captures user behavior and generates data on it. We often assume that instrumentation is straightforward and the complexity lies in what comes further (building data pipelines, deriving insights, building predictive models…), however if things aren’t properly defined at the collection step, all other steps of the data lifecycle are frail at best, unreliable at worst. We will deep-dive into instrumentation. Taking a real project, we’ll cover how it’s done at Shopify, particularly to capture behaviors cross-platform.

This presentation will focus on the architecture and best practices of batch processing in data warehouse ingestion across different datawarehouses. It will cover the design and implementation of batch pipelines, with an emphasis on incremental loads and idempotent syncs to ensure accurate and efficient data updates. Key strategies for optimizing incremental data processing and utilizing reusable code components will be explored to streamline pipeline development and maintenance. Practical examples will demonstrate how these techniques enable Churney, a startup that predicts customer lifetime value, to integrate seamlessly with various data warehouses, including BigQuery, Redshift, Athena, Snowflake, Databricks, and MaxCompute.

In the world of "AI" and language models, one size does not fit all. From nimble, small models to massive, powerhouse models like GPTs, the secret to unlocking their potential lies in how you prepare your data. The art and science behind data engineering still exists for different model scales. Eija-Leena will go through the critical steps for curating, cleaning, and transforming data for the multiple models they are using at SomeBuddy.

In today’s competitive business environment, retaining top talent while fostering employee growth is a critical challenge for organizations. “Streamlining Employee Progression and Retention with a Robust Data Pipeline” explores the integration of advanced data analytics to enhance employee development and reduce turnover rates. This approach utilizes a comprehensive data pipeline that collects, processes, and analyzes employee performance, engagement, and career growth metrics in real-time. By leveraging data-driven insights, companies can identify progression bottlenecks, create personalized development plans, and implement targeted retention strategies.

The vision promoted through conferences and social media is clear: organizations want to make decisions driven by knowledge through data, they are being promised reliable predictions about the behaviors of humans and systems. And yet, data teams and IT departments struggle to deliver on this.

In this presentation, Charlotte Møller will offer her perspective, an insider’s perspective, on what obstacles data teams face when given the task of building reliable data systems. In her view, the challenges go way beyond choosing the right tech stack for the project. Why is it that projects aiming at reducing complexity for others become so complex in themselves? What does turnover rates well above the average do to these teams? Does a degree in computer science provide the required knowledge to navigate the world of off-the-shelf products in the data industry?

This presentation will not provide you with a definite answer to these questions. They will, however, be reflected upon and put into perspective by someone who is working in this field on a daily basis. 

A discussion between practitioners in the field who have witnessed both successful and failed data projects. What does it take to succeed?

Panel Speakers:

• Rebecka Storm, Co-founder & CPO at Twirl
• Priyanka Narra, Senior Product Manager at Volvo Cars
• Gabriela Ayres, Global Credit Risk and Analytics Lead at Volvo Cars
  
  Moderated by: Lena Sundin, Freelance Data Engineer