Podd: DataTalks.Club

Trends in Data Engineering – Adrian Brudaru

7 mars 2025 | 57 min

In this podcast episode, we talked with Adrian Brudaru about the past, present and future of data engineering.

About the speaker:

Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.

As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.

0:00 Introduction to DataTalks.Club

1:05 Discussing trends in data engineering with Adrian

2:03 Adrian's background and journey into data engineering

5:04 Growth and updates on Adrian's company, DLT Hub

9:05 Challenges and specialization in data engineering today

13:00 Opportunities for data engineers entering the field

15:00 The "Modern Data Stack" and its evolution

17:25 Emerging trends: AI integration and Iceberg technology

27:40 DuckDB and the emergence of portable, cost-effective data stacks

32:14 The rise and impact of dbt in data engineering

34:08 Alternatives to dbt: SQLMesh and others

35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions

37:20 Audience questions: Career focus in data roles and AI engineering overlaps

39:00

The role of semantics in data and AI workflows

41:11 Focusing on learning concepts over tools when entering the field

45:15 Transitioning from backend to data engineering: challenges and opportunities

47:48 Current state of the data engineering job market in Europe and beyond

49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats

50:40 Suitability of these formats for batch and streaming workloads

52:29 Tools for streaming: Kafka, SQS, and related trends

58:07 Building AI agents and enabling intelligent data applications

59:09Closing discussion on the place of tools like DBT in the ecosystem

🔗 CONNECT WITH ADRIAN BRUDARU

Linkedin - / data-team Website - https://adrian.brudaru.com/ 🔗 CONNECT WITH DataTalksClub

Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/... Check other upcoming events - https://lu.ma/dtc-events LinkedIn - /datatalks-club Twitter - /datatalksclub Website - https://datatalks.club/

Competitive Machine Leaning And Teaching – Alexander Guschin

14 februari 2025 | 53 min

Redefining AI Infrastructure: Open-Source, Chips, and the Future Beyond Kubernetes – Andrey Cheptsov

31 januari 2025 | 57 min

In this podcast episode, we talked with Andrey Cheptsov about The future of AI infrastructure.

About the Speaker:

Andrey Cheptsov is the founder and CEO of dstack, an open-source alternative to Kubernetes and Slurm, built to simplify the orchestration of AI infrastructure. Before dstack, Andrey worked at JetBrains for over a decade helping different teams make the best developer tools.

During the event, the guest, Andrey Cheptsov, founder and CEO of dstack, discussed the complexities of AI infrastructure. We explore topics like the challenges of using Kubernetes for AI workloads, the need to rethink container orchestration, and the future of hybrid and cloud-only infrastructures. Andrey also shares insights into the role of on-premise and bare-metal solutions, edge computing, and federated learning.

00:00 Andrey's Career Journey: From JetBrains to DStack

5:00 The Motivation Behind DStack

7:00 Challenges in Machine Learning Infrastructure

10:00 Transitioning from Cloud to On-Prem Solutions

14:30 Reflections on OpenAI's Evolution

17:30 Open Source vs Proprietary Models: A Balanced Perspective

21:01 Monolithic vs. Decentralized AI businesses

22:05 The role of privacy and control in AI for industries like banking and healthcare

30:00 Challenges in training large AI models: GPUs and distributed systems

37:03 DeepSpeed's efficient training approach vs. brute force methods

39:00 Challenges for small and medium businesses: hosting and fine-tuning models

47:01 Managing Kubernetes challenges for AI teams

52:00 Hybrid vs. cloud-only infrastructure

56:03 On-premise vs. bare-metal solutions

58:05 Exploring edge computing and its challenges

🔗 CONNECT WITH ANDREY CHEPTSOV

Twitter - / andrey_cheptsov

Linkedin - / andrey-cheptsov

GitHub - https://github.com/dstackai/dstack/

Website - https://dstack.ai/

🔗 CONNECT WITH DataTalksClub

Join DataTalks.Club:⁠⁠⁠https://datatalks.club/slack.html⁠⁠⁠

Our events:⁠⁠⁠https://datatalks.club/events.html⁠⁠⁠

Datalike Substack -⁠⁠⁠https://datalike.substack.com/⁠⁠⁠

LinkedIn:⁠⁠⁠ / datatalks-club ⁠

Linguistics and Fairness - Tamara Atanasoska

17 januari 2025 | 53 min

In this podcast episode, we talked with Tamara Atanasoska about building fair AI systems.

About the Speaker:Tamara works on ML explainability, interpretability and fairness as Open Source Software Engineer at probable. She is a maintainer of fairlearn, contributor to scikit-learn and skops. Tamara has both computer science/ software engineering and a computational linguistics(NLP) background.During the event, the guest discussed their career journey from software engineering to open-source contributions, focusing on explainability in AI through Scikit-learn and Fairlearn. They explored fairness in AI, including challenges in credit loans, hiring, and decision-making, and emphasized the importance of tools, human judgment, and collaboration. The guest also shared their involvement with PyLadies and encouraged contributions to Fairlearn.

00:00 Introduction to the event and the community

01:51 Topic introduction: Linguistic fairness and socio-technical perspectives in AI

02:37 Guest introduction: Tamara’s background and career

03:18 Tamara’s career journey: Software engineering, music tech, and computational linguistics

09:53 Tamara’s background in language and computer science

14:52 Exploring fairness in AI and its impact on society

21:20 Fairness in AI models26:21 Automating fairness analysis in models

32:32 Balancing technical and domain expertise in decision-making

37:13 The role of humans in the loop for fairness

40:02 Joining Probable and working on open-source projects

46:20 Scopes library and its integration with Hugging Face

50:48 PyLadies and community involvement

55:41 The ethos of Scikit-learn and Fairlearn

🔗 CONNECT WITH TAMARA ATANASOSKA

Linkedin - https://www.linkedin.com/in/tamaraatanasoska

GitHub- https://github.com/TamaraAtanasoska

🔗 CONNECT WITH DataTalksClub

Join DataTalks.Club:⁠⁠https://datatalks.club/slack.html⁠⁠

Our events:⁠⁠https://datatalks.club/events.html⁠⁠

Datalike Substack -⁠⁠https://datalike.substack.com/⁠⁠

LinkedIn:⁠⁠ / datatalks-club

Career choices, transitions and promotions in and out of tech - Agita Jaunzeme

10 januari 2025 | 55 min

In this podcast episode, we talked with Agita Jaunzeme about Career choices, transitions and promotions in and out of tech.

About the Speaker:

Agita has designed a career spanning DevOps/DataOps engineering, management, community building, education, and facilitation. She has worked on projects across corporate, startup, open source, and non-governmental sectors. Following her passion, she founded an NGO focusing on the inclusion of expats and locals in Porto. Embodying the values of innovation, automation, and continuous learning, Agita provides practical insights on promotions, career pivots, and aligning work with passion and purpose.

During this event, discussed their career journey, starting with their transition from art school to programming and later into DevOps, eventually taking on leadership roles. They explored the challenges of burnout and the importance of volunteering, founding an NGO to support inclusion, gender equality, and sustainability. The conversation also covered key topics like mentorship, the differences between data engineering and data science, and the dynamics of managing volunteers versus employees. Additionally, the guest shared insights on community management, developer relations, and the importance of product vision and team collaboration. 0:00 Introduction and Welcome 1:28 Guest Introduction: Agita’s Background and Career Highlights 3:05 Transition to Tech: From Art School to Programming 5:40 Exploring DevOps and Growing into Leadership Roles 7:24 Burnout, Volunteering, and Founding an NGO 11:00 Volunteering and Mentorship Initiatives 14:00 Discovering Programming Skills and Early Career Challenges 15:50 Automating Work Processes and Earning a Promotion 19:00 Transitioning from DevOps to Volunteering and Project Management 24:00 Managing Volunteers vs. Employees and Building Organizational Skills 31:07 Personality traits in engineering vs. data roles 33:14 Differences in focus between data engineers and data scientists 36:24 Transitioning from volunteering to corporate work 37:38 The role and responsibilities of a community manager 39:06 Community management vs. developer relations activities 41:01 Product vision and team collaboration 43:35 Starting an NGO and legal processes 46:13 NGO goals: inclusion, gender equality, and sustainability 49:02 Community meetups and activities 51:57 Living off-grid in a forest and sustainability 55:02 Unemployment party and brainstorming session 59:03 Unemployment party: the process and structure

🔗 CONNECT WITH AGITA JAUNZEME Linkedin - /agita

🔗 CONNECT WITH DataTalksClub Join DataTalks.Club: ⁠https://datatalks.club/slack.html⁠ Our events: ⁠https://datatalks.club/events.html⁠ Datalike Substack - ⁠https://datalike.substack.com/⁠ LinkedIn: ⁠ / datatalks-club

Career advice, learning, and featuring women in ML and AI - Isabella Bicalho

13 december 2024 | 55 min

In this podcast episode, we talked with Isabella Bicalho about Career advice, learning, and featuring women in ML and AI.

About the Speaker:

Isabella is a Machine Learning Engineer and Data Scientist with three years of hands-on AI development experience. She draws upon her early computational research expertise to develop ML solutions. While contributing to open-source projects, she runs a newsletter dedicated to showcasing women's accomplishments in data science.

During this event, the guest discussed her transition into machine learning, her freelance work in AI, and the growing AI scene in France. She shared insights on freelancing versus full-time work, the value of open-source contributions, and developing both technical and soft skills. The conversation also covered career advice, mentorship, and her Substack series on women in data science, emphasizing leadership, motivation, and career opportunities in tech. 0:00 Introduction 1:23 Background of Isabella Bicalho 2:02 Transition to machine learning 4:03 Study and work experience 5:00 Living in France and language learning 6:03 Internship experience 8:45 Focus areas of Inria 9:37 AI development in France 10:37 Current freelance work 11:03 Freelancing in machine learning 13:31 Moving from research to freelancing 14:03 Freelance vs. full-time data science 17:00 Finding first freelance client 18:00 Involvement in open-source projects 20:17 Passion for open-source and teamwork 23:52 Starting new projects 25:03 Community project experience 26:02 Teaching and learning 29:04 Contributing to open-source projects 32:05 Open-source tools vs. projects 33:32 Importance of community-driven projects 34:03 Learning resources 36:07 Green space segmentation project 39:02 Developing technical and soft skills 40:31 Gaining insights from industry experts 41:15 Understanding data science roles 41:31 Project challenges and team dynamics 42:05 Turnover in open-source projects 43:05 Managing expectations in open-source work 44:50 Mentorship in projects 46:17 Role of AI tools in learning 47:59 Overcoming learning challenges 48:52 Discussion on substack 49:01 Interview series on women in data 50:15 Insights from women in data science 51:20 Impactful stories from substack 53:01 Leadership challenges in projects 54:19 Career advice and opportunities 56:07 Motivating others to step out of comfort zone 57:06 Contacting for substack story sharing 58:00 Closing remarks and connections

🔗 CONNECT WITH ISABELLA BICALHO Github: github https://github.com/bellabf LinkedIn: / isabella-frazeto

🔗 CONNECT WITH DataTalksClub Join DataTalks.Club: https://datatalks.club/slack.html Our events: https://datatalks.club/events.html Datalike Substack - https://datalike.substack.com/ LinkedIn: / datatalks-club

AI in Industry: Trust, Return on Investment and Future - Maria Sukhareva

6 december 2024 | 53 min

Large Hadron Collider and Mentorship – Anastasia Karavdina

22 november 2024 | 54 min

MLOps as a Team - Raphaël Hoogvliets

8 november 2024 | 56 min

Using Data to Create Liveable Cities - Rachel Lim

1 november 2024 | 46 min

We talked about:

00:00 DataTalks.Club intro 01:56 Using data to create livable cities 02:52 Rachel's career journey: from geography to urban data science 04:20 What does a transport scientist do? 05:34 Short-term and long-term transportation planning 06:14 Data sources for transportation planning in Singapore 08:38 Rachel's motivation for combining geography and data science 10:19 Urban design and its connection to geography 13:12 Defining a livable city 15:30 Livability of Singapore and urban planning 18:24 Role of data science in urban and transportation planning 20:31 Predicting travel patterns for future transportation needs 22:02 Data collection and processing in transportation systems 24:02 Use of real-time data for traffic management 27:06 Incorporating generative AI into data engineering 30:09 Data analysis for transportation policies 33:19 Technologies used in text-to-SQL projects 36:12 Handling large datasets and transportation data in Singapore 42:17 Generative AI applications beyond text-to-SQL 45:26 Publishing public data and maintaining privacy 45:52 Recommended datasets and projects for data engineering beginners 49:16 Recommended resources for learning urban data science

About the speaker:

Rachel is an urban data scientist dedicated to creating liveable cities through the innovative use of data. With a background in geography, and a masters in urban data science, she blends qualitative and quantitative analysis to tackle urban challenges. Her aim is to integrate data driven techniques with urban design to foster sustainable and equitable urban environments.

Links: - https://datamall.lta.gov.sg/content/datamall/en/dynamic-data.html 00:00 DataTalks.Club intro 01:56 Using data to create livable cities 02:52 Rachel's career journey: from geography to urban data science 04:20 What does a transport scientist do? 05:34 Short-term and long-term transportation planning 06:14 Data sources for transportation planning in Singapore 08:38 Rachel's motivation for combining geography and data science 10:19 Urban design and its connection to geography 13:12 Defining a livable city 15:30 Livability of Singapore and urban planning 18:24 Role of data science in urban and transportation planning 20:31 Predicting travel patterns for future transportation needs 22:02 Data collection and processing in transportation systems 24:02 Use of real-time data for traffic management 27:06 Incorporating generative AI into data engineering 30:09 Data analysis for transportation policies 33:19 Technologies used in text-to-SQL projects 36:12 Handling large datasets and transportation data in Singapore 42:17 Generative AI applications beyond text-to-SQL 45:26 Publishing public data and maintaining privacy 45:52 Recommended datasets and projects for data engineering beginners 49:16 Recommended resources for learning urban data science Join our slack: https: //datatalks.club/slack.html

DataTalks.Club 4th Anniversary AMA Podcast – Alexey Grigorev and Johanna Bayer

26 oktober 2024 | 54 min

Human-Centered AI for Disordered Speech Recognition - Katarzyna Foremniak

10 oktober 2024 | 48 min

DataOps, Observability, and The Cure for Data Team Blues - Christopher Bergh

15 augusti 2024 | 54 min