Bear with me… when I start writing a new piece I often have a goal in mind in terms of the message I want to deliver. But right now I have an idea going round in my head that I can’t quite pin down, so this this may turn into a bit of a slightly incohesive ramble.
It started with some interaction online around Microsoft’s new Data Platform Fabric and me challenging someone who was dismissing it as a low-code tool and not fit for proper data engineering. I didn’t agree with this view. Fabric isn’t all clicky-clicky-draggy-droppy. If you want to write a code first, Spark heavy implementation on Fabric, you absolutely can.
But then I got hung up on a thought… Why did this seem to matter to me so much?
I received some feedback at work recently calling out the fact that I can be quite self-effacing when it comes to my credentials as a Data Engineer. My background is as a full-stack Business Intelligence developer. Back in the day I looked after the full data lifecycle, ingesting data into the platform, modelling it and performing the transformation to get it in to a fit shape for analytics, as well as building and maintaining the semantic model and even building reports and dashboards.
Translated to technologies, I was a Microsofty using SQL Server, with SQL Server Server Integration Service (SSIS), SQL Server Analysis Services (SSAS) and SQL Server Reporting Services (SSRS).
At this point in my career I was considered a specialist and would describe myself as being a data professional.
The world has moved on. Technology has evolved and we’ve also seen roles and responsibilities change. Instead of the “jack of all trades” BI role, we increasingly see specialists emerging in more focussed roles such as Data Engineer, Analytics Engineer and Data Analyst.
I plant myself firmly in the Analytics Engineering camp. My skillset focusses mostly on data modelling and turning that into usable semantic models using Power BI.
Although my understanding of the concepts in Data Engineering remain, I’ve been less hands on for a long while and the move to code first software engineering style approaches isn’t a journey I really went on.
When I did use SSIS back in the day (and I’d describe myself as a competent as opposed to proficient user) I considered myself a “proper” developer. But thinking about it, SSIS is still pretty much a clicky-clicky-draggy-droppy product. It is perhaps less accessible then the current crop of citizen developer tools, but the level of skill to use it still isn’t that high. So why do I find myself turning my nose up at people taking self-service, low code approaches with the new crop of tools?
It’s because I’m a data snob isn’t it.
And I think that’s an undesirable trait. With some self reflection, I think I need to get better at accepting that there are multiple approaches to a problem.
The thing is, I see this data snobbery cropping up in lots of scenarios.
I see myself as a data professional, and take pride in the industry I work in. But we live in the age of information. These days EVERYONE is a data professional.
It feels like a fine line to tread in terms of being vocal about best practice versus gatekeeping of a particular skill. Do businesses depend on well crafted, highly efficient pipelines creating perfectly curated data sets and highly performant dashboards that follow recommended viz practices? Not really… they care about getting the data they need to make the decisions required to achieve their goals. If that’s a spreadsheet downloaded directly from an ERP system, does the CEO care? Nope.
Does being able to write great Python/SQL/DAX/AN Other language make your insights any better than Dave in finance’s hand cranked Excel chart that’s been screenshotted into Powerpoint? Surely not.
So am I dismissing the concept of data specialists? That would be extremely self-defeating and put me out of a job. There’s definitely still room to help craft more scalable, streamlined and automated approaches to getting those insights.
So what am I trying to say?
I think I’m trying to encourage people to be kind. That we all start somewhere, that the value of insight shouldn’t be predicated on the complexity of it’s underlying mechanics, that perfect shouldn’t be the enemy of good enough and that we should be less dismissive of approaches that don’t necessarily fit our perceptions of optimal.
I’m still not sure I’m making a particular coherent conclusion here. I guess maybe that low-code v pro-code or citizen developer v data pro set-ups should learn to co-exist better. And both crowds should be more accepting of each other. As practitioners we should use best practice as a tool to educate rather than a stick to beat people with.
I don’t feel as though I have the answers on how to do this effectively. But I’m hoping that acknowledging it is at least a first step on a journey for it to be something I get better at.
One last thing to share, as being reminded of it again recently is certainly one of the things that got my cogs grinding on this topic, is this excellent article from Kurt Buhler, which gives more food for thought on the be more kind front.
What are your thoughts? Can you sometimes be a data snob too? Is data snobbery ever a good thing? Or is it something we should make a conscious effort to stamp out?
2 Comments
Donald Parish · March 18, 2024 at 1:41 pm
Well said. Similar to snobbery vs VBA programs. Use right tool for the right job, and the right person.
Kristoffer Absalonsen · March 25, 2024 at 7:49 pm
Hear, hear! Let’s more the focus on the outcomes of solutions than how we get there. I think the data snobbery roadblocks the way to data democratization. As data professionals we want people to be more data driven, but only in the right way. Our way. Fit into our construct of managed self-serve dictatorship and you will succeed.