AI Engineering Requires a Different Mindset

Recently, after launching a couple of AI agents into production, I realized something: AI learning and development requires a very different mindset from almost any technology we have worked with before.

For years, the traditional approach to technology was relatively straightforward:

Learn a technology → use it to build a solution → become experienced with it → maintain and improve that solution over time.

We often invested years in a technology stack. We became experts in it, built our solutions around it, and expected that investment to remain valuable for a long time.

AI is changing that model.

AI is not just another technology to learn. It is part of a much faster evolution of IT itself.

Today, engineers need to learn faster, experiment faster, and become comfortable integrating tools and technologies they may have never touched before.

You might learn a new model today, integrate a new framework tomorrow, build an agent next week, deploy it, improve it, discover a limitation, and then replace part—or even all—of the solution a few weeks later.

And that's not failure.

That's the new way of building.

The goal is no longer to stay loyal to a particular technology. The goal is to achieve the business outcome.

If a technology is helping you achieve that goal, use it.

If you discover a better technology, integrate it.

If your solution has fundamental limitations, don't be afraid to destroy it and rebuild it.

Move on.

Move faster.

Because while you are trying to protect the solution you built six months ago, someone else may be building a better solution today.

The competitive advantage is no longer simply knowing a technology deeply. It is the ability to learn, experiment, integrate, deploy, evaluate, and rebuild—again and again.

I believe this is becoming one of the most important skills for engineers working with AI.

You need to be willing to upgrade your solution within months—or sometimes within weeks.

You need to be willing to throw away something you spent months building if it no longer serves the business goal.

You need to be able to rebuild a solution 10 or even 50 times faster than you could have done before.

That mindset is uncomfortable.

But it is also incredibly exciting.

I remember when I was in Singapore, one of my colleagues, an engineer, used to say:

"Don't ask me if it's possible. Just tell me what to build. I am IT God—I will build it."

At the time, I thought it was a great engineering attitude.

Today, I see even more truth in it.

That's where every engineer working with AI needs to get.

Not necessarily knowing every answer.

Not knowing every tool.

Not being attached to one technology.

But having the confidence to say:

Tell me what the business needs.
I'll figure out how to build it.

And if the technology changes tomorrow?

I'll learn it.
I'll integrate it.
I'll build it again.

That's the AI engineering mindset.

NotebookLM – Turning Information Overload into Insights

 

Have you ever been buried under a mountain of documents, recordings, and project notes, wishing there was a magic button to just tell you the important parts?

That has been my struggle for a long time. During Knowledge Transfer (KT) sessions in past projects, I often ended up with piles of files—videos, audio recordings, and lengthy documentation. And when the time came to find one small but crucial piece of information? The traditional way meant endless scrolling, skimming, or—worse—rewatching hours of video just to locate a single answer.

Now, picture those massive Confluence spaces or giant repositories where you only need a quick detail. Instead of a five-minute task, it turns into a few hours of searching. Frustrating, right?

That’s why discovering NotebookLM felt like a game-changer. Built on Gemini, NotebookLM is like having a smart AI research assistant right beside you. It doesn’t just summarize your documents or pull out precise answers—it makes working with knowledge feel effortless.

And here’s the part that blew me away: it can actually transform your content into podcast-style audio. Not the boring line-by-line text-to-speech, but an interactive, conversational podcast that feels like someone is walking you through the key insights. Imagine listening to your project documents while commuting or sipping coffee—suddenly, knowledge becomes portable and engaging.

For me, NotebookLM isn’t just another tool—it’s a smarter way to work with knowledge, saving time, reducing frustration, and even making the process enjoyable.

 

Here I have 20 Page PDF for “Build a modern, distributed Data Mesh with Google Cloud” I am going to create a NotbookLM and upload this document anow I have every thing ready a audio , a prodcast and summart and the most interesting thing is I am bale to ask question based on this document.

Here is practical example :

Here is the video created by this toom about data mesh based on my document. 

Here is Mind Map The most interesting part : 

Data at Your Fingertips: AI Makes It Instant

 

🚀 AI is evolving faster than any tech I've seen before — and it's changing the game.

As a Senior Data Engineer and an AI Engineer enabler, I’ve recently implemented several AI agents that have completely transformed the way we interact with information.

📄 Imagine this:
You have thousands of PDFs—agreements, contracts, invoices, resumes—and instead of searching through folders and files, you just ask:
"What's the notice period in our vendor agreements?" or
"Do we have a candidate skilled in Rust and Kubernetes?"
And you get the answer in seconds. No stress. No endless clicking. No guessing where you saved what.

These are not ideas. They're real solutions we’ve already put in place.

💡 Just recently, I built a system that automated the processing of massive documentation, making it instantly searchable and intelligent. HR, Legal, Finance—any department can now query their data like talking to a colleague.

🧠 And it's not just documents:
We’ve also experimented with AI agents on top of our databases—eliminating the need to write SQL for many use cases. Analysts no longer need to remember joins, filters, or aggregations. Instead, they focus on connecting insights, not building queries.
This is also opening the door for non-technical teams to engage with data more confidently—building a stronger data culture across the organization.

🔍 The days of search and read are numbered. Just like the pen started disappearing from desks, manual navigation through data is vanishing too.

I can't stop diving deeper into AI—and I’m excited to keep building what’s next. 

#AI #DataEngineering #IntelligentAutomation #AITools #GenerativeAI #AIEngineering #FutureOfWork #SeniorDataEngineer #AIEnabler

 

The Growing Importance of Data Professionals

Key Skills and the Path to Becoming a Data Person

Data has become one of the most critical assets for organizations across industries. As the volume of data continues to grow exponentially, the demand for skilled professionals who can manage, analyze, and leverage this data has skyrocketed. These professionals, often referred to as "data persons," play a pivotal role in every organization, bridging the gap between complex data and strategic business decisions.

In recent years, the demand for data professionals has increased dramatically, and this trend will only continue as data grows every second. In this blog, I'll share insights from my experience and provide guidance on how to become a successful data person, along with the essential skills and knowledge that an engineer needs to excel in this field.

The Evolving Role of a Data Professional

Through my career, I’ve noticed that a data person is not just an engineer focused on raw technical skills; they also need to have a solid understanding of the end-to-end business processes, much like a business analyst. In many ways, the data person acts as a bridge between the technical data world and the leadership team, including C-level executives. This makes their role not only technical but also strategic and vital for the organization's success.

The Key Skills Every Data Person Needs

From my experience on interview panels, I’ve observed that many candidates often overlook a critical skill: business understanding. While technical expertise is essential, a data person must understand the business context and objectives to ensure that their work aligns with the company's goals.

Another foundational skill that many candidates miss, even at senior levels, is proficiency in SQL. Whether you're a data engineer, data scientist, or data analyst, SQL remains one of the most important tools in the data person's toolkit. Along with SQL, a basic understanding of programming—particularly Python—has become a must-have skill.

The Rise of the Full-Stack Data Person

The role of a data professional has evolved dramatically in recent years. Industries no longer expect a data person to specialize in just one aspect of data management. Today, the expectation is for a full-stack data professional who can handle a range of tasks, including:

• Data Engineering: Bringing data from diverse sources, managing data pipelines, and ensuring data integrity.
• Data Analysis: Analyzing data and ensuring its accuracy to make informed decisions.
• Machine Learning: Collaborating with data scientists to apply machine learning models and share insights.
• Data Modeling: Designing data models, at least for staging purposes.
• Cloud Data Tools: Proficiency in cloud-based data tools (e.g., AWS, Azure, Google Cloud).
• CI/CD for Data: Understanding data CI/CD tools like Terraform to automate and streamline data workflows.

This broad set of skills makes the role of the data person more dynamic and challenging, but also more rewarding.

How Technology is Easing the Data Person’s Journey

While the expectations have grown, the tools and technologies available to data professionals have also improved significantly. Modern data engineering tools, fully managed data services, and comprehensive tool documentation have streamlined the process, making it easier for data professionals to do their jobs more efficiently. These advancements have simplified tasks such as data integration, query optimization, and troubleshooting, which once took up significant amounts of time and effort.

Conclusion: The Future of Data Professionals

The role of the data person is more crucial than ever. As organizations continue to rely on data to drive business decisions, the demand for skilled data professionals will only grow. To succeed in this field, it’s important not to overlook the foundational skills of business understanding and SQL. Additionally, given the industry's evolving expectations, becoming a full-stack data person is key to staying relevant in the future.

The landscape of data has changed dramatically, and while the challenges have increased, the tools and resources available to data professionals have made their work more efficient and impactful than ever before. As a result, pursuing a career as a data person is not only a great opportunity today but will continue to be one of the most in-demand roles in the future.

About Me

Hi, I’m Vikas Kumar Pathak , a Data Architect and Data Engineer with 14 years of experience crafting innovative, scalable, and secure data solutions across diverse industries including Finance, Retail, Education, and Customer Loyalty.

I specialize in designing and leading the development of cloud-based data platforms, leveraging tools such as Google Cloud Platform (GCP), AWS, BigQuery, and Redshift. My expertise spans the entire data lifecycle—from architecture and engineering to advanced analytics and machine learning. I’m passionate about building automated, high-performance data systems that drive business growth and decision-making.

Throughout my career, I’ve led teams in designing and implementing complex data solutions, managed large-scale data transformations, and contributed to SaaS-based platforms. I thrive in roles that require both technical proficiency and strategic oversight, ensuring data solutions align with business objectives while maintaining security and scalability.

Currently, as a Sr. Cloud Data Architect & Data Engineer at GIIR Germany GmbH, I’m leading the creation of an enterprise-level Customer Data Platform (CDP), focusing on data security, automation, and seamless integration to deliver actionable customer insights.

I’m always exploring new ways to improve data systems and drive insights, and I’m excited about the future of data-driven decision-making.

• Email : hatch.vikas2011@gmail.com
• LinkedIn  :   https://www.linkedin.com/in/vikas-kumar-pathak-89876a47/
• Git :    https://github.com/vikaskpathak
• Medium contribution : https://medium.com/@hatch.vikas2011

Data Pipeline as a Service (DPaaS)

Understanding Data Pipeline as a Service (DPaaS): 

I define "Data Pipeline as a Service" (DPaaS) as a concept primarily employed when constructing data pipelines for SaaS applications, facilitating the transfer of data from a SaaS-based OLTP system to an OLAP system.

Data Pipeline as a Service (DPaaS) offers a streamlined solution for building and managing data pipelines, particularly in multi-tenant SaaS environments. It enables the reuse of the same pipeline logic for different tenant while ensuring strict separation of data flows to prevent cross-tenant data leakage—a critical challenge in multi-tenant architectures.

The Multi-Tenant Data Challenge

Consider a common scenario: Your SaaS application needs to build a data warehouse. While the data transfer logic (ETL/ELT process) may be consistent across tenant , merging the pipelines can introduce significant risks:

1. Data Leakage: Without robust isolation, there’s a chance that data from one tenant could inadvertently flow into another tenant's pipeline, creating security and compliance issues.
2. Pipeline Dependencies: A failure in one tenant’s pipeline could bring down processes for all tenants, causing widespread disruptions.
3. Maintenance Complexity: Optimizing or updating pipelines can be error-prone, as it requires modifying logic for each tenant. Missing updates for even a single tenant could lead to inconsistent performance or failures.

Most of existing solutions often attempt to address these challenges by tenant code as parameter throughout the ETL process. However, this approach is cumbersome, error-prone, and does not fully mitigate the risk of cross-tenant data leaks.

How DPaaS Solves the Problem

DPaaS introduces a tenant-specific approach to pipeline creation. Instead of managing a single shared pipeline for all tenants, DPaaS allows you to:

• Isolate Pipelines: Build a shared codebase but instantiate a separate pipeline for each tenant. This ensures complete data isolation and prevents cross-tenant leakage.
• Improve Reliability: If one tenant’s pipeline encounters an issue, it doesn’t impact others, ensuring seamless operation for unaffected tenant .
• Simplify Updates: Changes or optimizations are applied at the codebase level and propagated through individual pipeline instantiations. This reduces the risk of missing updates for specific tenants.
• Backfill and Retry : Backfill and retry easily possible for a specific tenant instead of running pipeline for all tenant.
• Tenant onboarding:  Onboarding new tenant with almost zero developer effort. Its just matter of adding one more tenant name in tenant list variable.

The Value for SaaS Companies

For SaaS product companies dealing with large-scale data, DPaaS offers a scalable and secure way to manage data pipelines. It eliminates the complexities and risks associated with multi-tenant ETL processes, ensuring compliance, operational reliability, and ease of maintenance—all while enabling consistent performance and scalability.

With DPaaS, you can focus on building efficient pipelines and serving your tenant , confident that their data remains secure and isolated.

"As a Service" Mindset : 

 One of the foundational principles for successfully implementing a DPaaS solution is adopting an "as a service" mindset. This approach ensures that teams think about problems from a service-oriented perspective, focusing on scalability, flexibility, and tenant-agnostic solutions. I have details in my one of post here :

https://sqlvikas.blogspot.com/2024/12/embracing-dpaas-mindset.html

Embracing the "DPaaS" Mindset

 

Embracing the "As a Service" Mindset

One of the foundational principles for successfully implementing a DPaaS solution is adopting an "as a service" mindset. This approach ensures that teams think about problems from a service-oriented perspective, focusing on scalability, flexibility, and tenant-agnostic solutions.

When dealing with tenant-specific change requests, it’s crucial for data engineers to approach the challenge with this service mindset. Instead of immediately resorting to quick fixes or hardcoding solutions—like embedding a tenant code directly into the pipeline—they should explore alternatives that align with the service-first philosophy.

Example Scenario

Imagine a tenant requests a new feature or customization in their pipeline. A traditional approach might involve hardcoding the tenant’s unique identifiers or logic directly into the shared pipeline. While this might work in the short term, it introduces risks like:

• Complicating the pipeline logic over time.
• Increasing the chances of cross-tenant data leaks or errors.
• Making future updates harder to implement across multiple tenants.

Instead, the service mindset encourages engineers to ask:

• "Can we implement this change in a way that benefits all tenants?"
• "How can we integrate this feature without tenant-specific hardcoding?"
• "Does this approach scale as more tenants request similar features?"

By prioritizing reusable, tenant-agnostic solutions, engineers can maintain the integrity and scalability of the DPaaS architecture while still addressing tenant-specific needs.

The Service Mindset in Action

For instance, instead of embedding tenant-specific logic, you could leverage configuration-driven designs. This means creating a flexible pipeline framework where tenant-specific behaviors are defined by configurations stored in a metadata layer. Each pipeline instance pulls its configuration dynamically, ensuring isolation while avoiding custom hardcoding.

This mindset not only simplifies maintenance and updates but also ensures the pipeline remains robust and scalable as the number of tenants grows. By thinking of pipelines as a service rather than a one-off solution, teams can deliver a secure, efficient, and adaptable experience for all clients.