How to build a top data engineering team

Building a data engineering team that the business actually trusts is harder than it looks. I've seen teams with brilliant engineers ship pipelines that break every quarter, miss SLAs nobody knew existed, and produce numbers that don't match what finance expects. The problem isn't technical competence—it's everything around the code.

Here's what I've learned about building teams that ship reliable data infrastructure and earn the trust that comes with it.

Part 1: Team Building

Roles and responsibilities

A top data engineering team isn't just a collection of people who can write SQL and Python. You need clear roles, and those roles need to make sense for your scale.

At minimum, you need:

• Pipeline owners — People who own end-to-end data flows. Not just the code, but the SLA, the data quality, the stakeholder communication, everything. If a pipeline breaks at 3am, they know because they set up the monitoring.
• Platform engineers — People who build the infrastructure other engineers use: orchestration, CI/CD, metadata stores, observability tooling. If your data engineers are manually SSH-ing into boxes, you don't have this role and you need it.
• Data modelers — People who design how data should be structured. At small scale this can be the same person as the pipeline owner. At large scale it can't be.

The key insight: ownership has to be end-to-end. If the person who writes the pipeline isn't the person who gets paged when it breaks, your incentives are wrong and your pipelines will be fragile.

Hiring bar

The hiring bar for data engineering is not "can you write a Spark job." It's "can you own a system that processes money-critical data and debug it when it breaks in production at 2am."

What I look for:

• Systems thinking — Can they reason about failure modes? Do they think about idempotency, retries, backpressure, data skew? Can they debug a pipeline that's slow without just adding more nodes?
• Ownership mentality — Have they owned production systems before? Do they talk about monitoring, alerting, runbooks? Or do they just talk about the happy path?
• Communication — Can they explain technical trade-offs to non-technical stakeholders? Can they write documentation that junior engineers can actually use?
• Pragmatism — Do they reach for the simplest solution or the fanciest one? A senior engineer who suggests a cron job when a cron job is enough is worth their weight in gold.

Red flags: people who can't explain their past work in detail, people who've never been on-call, people who've never had to clean up their own technical debt.

Ownership and on-call

If your data engineers aren't on-call, your pipelines will be unreliable. Full stop.

On-call does two things: it creates a feedback loop that makes engineers care about reliability, and it forces you to invest in the tooling that makes on-call bearable. Good monitoring, good alerting, good runbooks—none of this happens unless the people writing the code are the ones waking up at 3am when it breaks.

The business will trust your team when your team takes ownership of failures. This means:

• Transparent incident postmortems
• SLAs with teeth (and consequences when you miss them)
• Proactive communication when things go wrong
• A culture where "I don't know but I'll find out" is acceptable and "it works on my machine" is not

Code review and standards

Code review is where your team's standards get enforced. If you're not reviewing data pipelines with the same rigor you review application code, you're building technical debt.

What I look for in code review:

• Is the pipeline idempotent? Can you run it twice and get the same result?
• How does it handle failures? What happens if the source is late? What happens if it's malformed?
• Is it testable? Are there tests?
• Is there monitoring? Are there alerts? Is there a runbook?
• Can someone else understand this code in six months?

If the answer to any of these is "no," the code doesn't ship.

Documentation and data contracts

Documentation isn't optional. If your pipelines aren't documented, they're not maintainable, and if they're not maintainable, they'll rot.

What you need:

• Pipeline documentation — What does this pipeline do? What's the SLA? Who owns it? Where does the data come from? Where does it go? What are the failure modes?
• Data contracts — What's the schema? What are the constraints? What happens if the schema changes? Who's allowed to change it? This should be in code (dbt models, Great Expectations suites, Protobuf schemas) not Word docs.
• Runbooks — When this pipeline breaks, what do you do? What are the common failure modes? How do you debug them? How do you recover?

The business trusts teams that document their work because it signals that the team expects their systems to outlive any individual contributor.

Part 2: SAP Data Services Architecture

Now let's talk about SAP Data Services specifically. If you're building ETL pipelines in an enterprise environment, there's a good chance you're using it. It's powerful, it's mature, and it's deeply misunderstood by people who haven't spent time in the guts of it.

How SAP Data Services works under the hood

SAP Data Services is a suite of ETL tools built around a metadata-driven architecture. Here's the high-level breakdown:

Repository — This is where all your metadata lives: job definitions, data flows, transformations, connections, table definitions. It's a relational database (usually Oracle or SQL Server) that stores everything about your ETL environment. A local repository holds your working objects; for team development and versioning you can add a central repository with check-in and check-out, but it is not Git, so many teams also export their objects (ATL files) into Git for real change history.

Designer — This is the IDE where you build data flows, work flows, and jobs. It's a thick client that connects to the repository. Think of it as Eclipse for ETL. You drag and drop transformations, configure them, and the Designer generates metadata that gets stored in the repository.

Job Server — This is the runtime engine that executes jobs. When you run a job, the Job Server starts a job engine process that reads the job's metadata from the repository, optimizes it into an execution plan, and runs it. The engine is where data gets read from sources, transformed, and loaded into targets.

Access Server — This is the server for real-time jobs. It receives requests (for example through web services), routes them to real-time services, and returns the response. It is not used for ordinary batch jobs. Pushdown and other optimization are done by the job engine at run time, covered below.

Data flows, work flows, and jobs

SAP DS has three levels of abstraction:

• Data flows — These are the atomic units of transformation. A data flow reads from one or more sources, applies transformations (filters, joins, aggregations, lookups), and writes to one or more targets. Data flows are where your actual ETL logic lives.
• Work flows — These are orchestration units. A work flow is a directed graph of data flows, scripts, and conditional logic. Think of it as a mini-DAG. You use work flows to chain data flows together, handle errors, and implement conditional branching.
• Jobs — These are the top-level execution units. A job is a work flow plus configuration: database connections, global variables, runtime parameters. When you schedule a job, you're scheduling a job, not a work flow or a data flow.

Engine and optimization

The SAP DS engine is smart about optimization. Here's what it does:

Pushdown optimization — If a transformation can be executed in the source or target database, the engine pushes it down. For example, if you have a Query transform that just filters rows, the engine generates a SQL WHERE clause and lets the database do the work. This is almost always faster than pulling data into the Job Server.

You can control pushdown behavior. Sometimes you want to force pushdown (e.g., when your database is more powerful than your Job Server). Sometimes you want to disable it (e.g., when you're doing complex transformations that the database can't handle efficiently).

Parallel execution — The Job Server can run multiple data flows in parallel if they're independent. This is controlled by the degree of parallelism setting. You can also partition data flows to process subsets of data in parallel (e.g., process each region in a separate thread).

Bulk loading — For large loads, the engine can use database-specific bulk loading APIs (e.g., Oracle SQL*Loader, SQL Server bulk insert). This is way faster than row-by-row inserts. You enable this in the target table configuration.

Part 3: Design Patterns and Best Practices

Stage → Transform → Load

The canonical pattern in SAP DS is stage-transform-load:

1. Stage — Pull raw data from the source into a staging area. This is a straight copy, no transformations. The goal is to get data into your environment as fast as possible and minimize the load on the source system.
2. Transform — Apply business logic, clean data, join dimensions, calculate metrics. This happens in your environment, not in the source.
3. Load — Write the transformed data to the target (data warehouse, data mart, operational data store).

Why this pattern? Because it decouples extraction from transformation. If your transformation fails, you don't need to re-extract. If your source system is slow, you minimize the extraction window. If you need to re-run the transformation with different logic, you have the raw data in staging.

Delta and CDC loading

Full loads are simple but expensive. For large tables, you want delta loading: only pull rows that changed since the last load.

SAP DS has two transforms for this:

Table_Comparison — Compares a source table to a target table and identifies new, changed, and deleted rows. This is useful when your source doesn't have change tracking. The downside: it requires reading both the source and target, so it's expensive for large tables.

History_Preserving — This is for slowly changing dimensions (SCD Type 2). It tracks historical changes by adding effective date and expiration date columns. When a row changes, the old row gets expired and a new row gets inserted.

For true CDC, you need a source system that tracks changes (e.g., Oracle GoldenGate, SQL Server Change Data Capture, a timestamp column). Then you can just pull rows where modified_date > last_load_date.

Here's a SQL pattern for delta merge:

-- Merge new and updated rows into target
MERGE INTO target_table t
USING staging_table s
ON t.primary_key = s.primary_key
WHEN MATCHED AND t.hash_value != s.hash_value THEN
  UPDATE SET 
    t.column1 = s.column1,
    t.updated_at = CURRENT_TIMESTAMP
WHEN NOT MATCHED THEN
  INSERT (primary_key, column1, created_at)
  VALUES (s.primary_key, s.column1, CURRENT_TIMESTAMP);

The hash value trick is important: instead of comparing every column, compute a hash of the row and compare that. Much faster.

Surrogate keys

Use surrogate keys for dimension tables. Don't use natural keys from the source system—they change, they're not guaranteed to be unique across sources, and they're often not integers (which means slower joins).

In SAP DS, you generate surrogate keys using the Key_Generation transform or a database sequence. Store the mapping (natural key → surrogate key) in the dimension table and use that for lookups.

Error handling and audit tables

Every data flow should have error handling. In SAP DS, you can do this with the Validation transform, which splits rows into pass and fail outputs so you can route failures to an error table, together with try/catch blocks at the work flow level for job-level failures.

Don't just log errors—capture the row that failed, the error message, and enough context to debug it. A good error table schema:

CREATE TABLE etl_errors (
  error_id INT PRIMARY KEY,
  job_name VARCHAR(255),
  data_flow_name VARCHAR(255),
  error_timestamp TIMESTAMP,
  error_message TEXT,
  source_row TEXT,  -- JSON or delimited string
  retry_count INT DEFAULT 0
);

Audit tables track job execution: when did the job run, how many rows were processed, did it succeed or fail? This is critical for troubleshooting and SLA tracking.

CREATE TABLE etl_audit (
  audit_id INT PRIMARY KEY,
  job_name VARCHAR(255),
  start_time TIMESTAMP,
  end_time TIMESTAMP,
  status VARCHAR(50),  -- SUCCESS, FAILURE, RUNNING
  rows_read INT,
  rows_written INT,
  rows_rejected INT
);

Reusable templates and parameterization

Don't copy-paste data flows. Build reusable templates using custom functions and parameterized jobs.

Custom functions — These are user-defined transforms you can reuse across data flows. For example, if you have a standard date formatting function or a standard address parsing function, wrap it in a custom function.

Global variables — Use global variables for environment-specific config (database connections, file paths, date ranges). This lets you promote the same job from dev to test to prod without changing the code.

Parameterized jobs — Pass parameters to jobs at runtime (e.g., load_date, region). This makes your jobs flexible and testable.

Naming conventions

Naming conventions are boring but essential. A good convention:

• Jobs: JOB_[source]_[target]_[frequency] (e.g., JOB_SALES_DWH_DAILY)
• Data flows: DF_[action]_[entity] (e.g., DF_LOAD_CUSTOMERS)
• Work flows: WF_[process] (e.g., WF_MONTHLY_CLOSE)
• Tables: [layer]_[entity] (e.g., STG_CUSTOMERS, DIM_CUSTOMERS)

Restartability and recovery

Your jobs should be restartable. If a job fails halfway through, you should be able to re-run it without corrupting data.

Patterns for restartability:

• Truncate and reload — For small tables, just truncate the target and reload. Simple, idempotent, but expensive for large tables.
• Delete and insert — Delete the rows you're about to load (e.g., DELETE WHERE load_date = CURRENT_DATE), then insert. Works if you're loading by partition.
• Staging table swap — Load into a staging table, then swap it with the production table (e.g., RENAME TABLE prod TO prod_old, staging TO prod). Atomic, but requires DDL privileges.

Logging

Built-in logging in SAP DS is decent but not great. You get logs in the Management Console, but they're not structured and they're hard to query.

Better: write your own logs to a database table. Use a Script transform to log key events (job start, job end, row counts, errors). Then you can query those logs, build dashboards, set up alerts.

Part 4: Automation via API

There are a few supported ways to drive SAP DS from outside its own scheduler. You can export an execution command for a job from the Management Console Administrator, which generates a launcher script you call from any scheduler. You can also use the web services exposed by the Management Console and Access Server (SOAP, described by a WSDL on your own server). The exact operations and endpoints depend on your version, so read the WSDL on your server rather than trusting a blog post, including this one.

Here is a generic Python wrapper around the exported launcher script. Paths, job name and arguments are placeholders:

import subprocess

LAUNCHER = "/path/to/exported/JOB_SALES_DWH_DAILY.sh"  # exported from the Management Console
GLOBAL_VARS = {"$G_LOAD_DATE": "2026-10-24"}              # global variables passed at runtime

def run_job(launcher: str, global_vars: dict[str, str]) -> int:
    cmd = [launcher]
    for name, value in global_vars.items():
        cmd += ["-G", f'"{name}={value}"']  # flag syntax depends on your export, check it
    result = subprocess.run(cmd, capture_output=True, text=True)
    print(result.stdout[-2000:])
    return result.returncode

if __name__ == "__main__":
    raise SystemExit(run_job(LAUNCHER, GLOBAL_VARS))

For the SOAP route, point a client such as zeep at the WSDL of your own server and call the batch job operations it lists. Keep credentials in a secret store, never in the script.

Use the API to build your own orchestration layer on top of SAP DS. Trigger jobs from Airflow, monitor them from Datadog, send alerts to Slack. The built-in scheduler works, but an external orchestrator gives you dependencies across systems, retries and one place for alerting.

Conclusion

Building a top data engineering team isn't about hiring rockstars or adopting the latest tools. It's about ownership, standards, communication, and reliability. The teams the business trusts are the teams that show up when things break, document their work, and deliver data on time.

And if you're working with SAP Data Services, understand the architecture. Learn the patterns. Build for restartability, auditability, and maintainability. Automate what you can. Document what you can't.

The best data engineering teams are boring. Their pipelines run on schedule. Their data is correct. Their documentation is up to date. Nobody talks about them because there's nothing to talk about—they just work.

That's the goal.