How to build a top data engineering team

I remember listening to a podcast where Marc Andreessen said that working at SpaceX is like being drowned in a zone of shocking competence. Everybody is just insanely good at what they do. For some reason, after that podcast, I became really interested in the idea of assembling a team of excellent engineers and building the culture to support them. Naturally, I started looking at my own team at work and evaluating us. On a scale of 1 to 10, how good a team are we? How good are the people on it? How good is our culture? The answers were not great. So I started asking myself why, and how we could close the gap. This led me to a few questions: What does a top engineer look like for our team? What does a perfect team look like? And what is the culture inside such a team?

1) What a top engineer looks like for our team

I have identified four areas in which an engineer on our team should excel to become a top data engineer:

• Attitude
• Business knowledge
• Data engineering concepts
• Mastery of the tools

Let’s dive into each of them:

• Attitude: An engineer should have a mindset of adding value and contributing. He should always ask himself: How can I help the team? How can I contribute? How can I make things better? He should have the desire and the agency to solve problems on his own initiative. He should not wait for tasks to be given to him, and he should not optimize for doing the least amount of work possible. He should feel responsible for his work: he should do his best to solve the issue at hand, even if it doesn’t directly depend on him. He must hold high standards for himself and for the work he does.
• Business knowledge: For data engineers, the business is at the core of everything we do. A top data engineer should have an excellent understanding of the business.
• Data engineering concepts: He must master the concepts and best practices of data engineering: data warehousing, data modeling, data pipeline design and debugging, SQL, data visualization, and communication.
• Mastery of the tools: He must master the tools he uses every day (SAP Data Services, SAP BI, Tableau, etc.).

I believe that when each of these elements is maxed out, we have a top engineer. And if everybody on the team is like that, we will produce more, and our work will be of higher quality.

2) Culture

Culture inside a company is the shared set of values, beliefs, attitudes, and unwritten rules that shape how people think, interact, and work within an organization. It’s the social operating system that dictates what is encouraged, tolerated, or discouraged.

If we want to get the best out of people, we need to create a culture that helps us do so. Here is what we can improve in ours:

• Taking initiative is not rewarded. If showing initiative only gets you more work and no recognition, and a mistake gets you blamed, then doing the minimum is the rational choice. This leads to situations where nobody wants to propose anything, for fear that it will only bring more work. We need to reward initiative from our engineers.
• Excellence must become the norm. The more we tolerate mediocrity and bad attitudes, the more they become the standard at which the entire team operates. And the more we are surrounded by excellent people doing excellent work, the more we are forced to raise our own standards. This creates a snowball effect where things just keep getting better.
• Engineers must own problems. Not tasks. Not building the pipeline, not building the dashboard, but the whole problem.
• People must be aligned behind a mission. We spend a considerable part of our lives at work. If people feel that what they do doesn’t really matter, and they can’t wait to go home because work feels like it is stealing their time, then somewhere we have failed to convey the mission behind everything we do.
• Speed is absolutely essential. The faster we move, the sooner we get feedback; the more often we update based on that feedback, the faster we learn. We can learn in one month what used to take us five.

Conclusion

Going back to that podcast, I don’t think SpaceX engineers are born different from us. They are good engineers placed in a culture that expects excellence, rewards initiative, and moves fast. Top engineers and a strong culture feed each other: the culture brings out the best in people, and excellent people raise the culture.

None of this will change overnight, and it doesn’t start with a big announcement. It starts with each of us: raising our own standards, owning the problems in front of us, and recognizing it when a teammate does the same. If we do that consistently, the next time we ask ourselves how good a team we are on a scale of 1 to 10, I believe the answer will be very different.

Jules Nembot