Category: Data analytics

As the dark winter of 2023 draws in and I come to the end of 6+ years as a freelance data analyst focused on social impact, now is as good a time as any for some reflection.

I’ve compiled some thoughts and highlights from recent conversations I’ve had with students and others in the social impact sector.

Hopefully, my position as an independent consultant working with charities large and small on a range of data analysis projects adds perspective and value.

What are the characteristics of a data-driven approach to social and humanitarian issues?

These days, everyone is obsessed with data but there is a big difference between a proper data-driven approach and collecting data for the sake of it.

In the non-profit sector, I’m aware of a lot of reporting back to different stakeholders (particularly to donors). It’s in the job description or organisational strategy. Data is collected to satisfy some fundraising obligation or regulatory requirements, often at great cost of time and effort. And it ends there.

A true data-driven approach is about people organically engaging with data and insight across the organisation, working to agreed measurement and evaluation frameworks. It is critical to take useful action or make optimisations to activities based on the observations. As I see it, that’s the only way to use data to inform decision making and improve organisational outcomes, i.e. to be “data-driven”. Of course, it’s easier said than done, but when it works it can have a huge impact.

Another background issue I often see is that data and insight is consigned to technical experts or highly performance-oriented functions like marketing or fundraising. For true impact, we need to embed the rigorous use of data with policy and programme delivery teams (as well as senior management and the board).

I think this is very much a work in progress across the sector, though I’ve seen encouraging signs in recent times. Different organisations will have their own host of challenges in this regard, not least with resourcing and technical expertise. One of my own major aims is to de-mystify data analytics and make it much more broadly accessible.

How do you see the future prospects and key roles for the development of this sector?

I think there is huge scope for data science and analytics to make even greater impact in the sector.

This has been my core mission for over 10 years now; things have been changing slowly. I would say the change is now accelerating. People are embracing data and tech more, even when they don’t have a lot of technical expertise themselves.

The two major, overarching opportunities right now I would say are:

1. Bringing data science and analytics to cross-functional teams and elevating its influence to non-technical parts of the organisation, and
2. Embracing the fact that AI and machine learning have now become mainstream to supercharge analytics efforts, by helping automate menial tasks and basic reporting.

The first opportunity is something that is happening slowly but steadily. I see much better data literacy in general compared with 5-10 years ago. Even when stakeholders have little data expertise, they tend to realise how crucial it is and are willing to learn more and outsource insight work as appropriate (whether externally or internally). I do still see a lot of daily struggle using and understanding data, which is a big barrier to useful insight that I hope we will soon overcome. “Data” is, essentially, just information.

Perhaps key to this is the second opportunity, the proliferation of AI and machine learning. As someone who began studying AI and Computer Science over 17 years ago(!) I’m slightly jaded by the hype around it all. However, I can see with the advent of generative AI and accessible, natural language interfaces and apps that a real step change is happening.

I think this is particularly good for tech and data analysis: so much time and energy is spent on just making things work – building, testing, QA – or producing highly repetitive reporting and dashboards.

I do believe that we will always need the human touch and cross-disciplinary understanding to review and refine work. However, AI can really shoulder some of that early to middle analysis workload, and help with automation and templates.

In terms of roles, there are a now many highly distinct disciplines in the sector, including:

• Data analyst
• Data scientist
• Monitoring, evaluation, and learning manager
• Research analyst
• Data engineer

We have now moved past the point where we can expect any one person to cover all bases (seriously, organisations of any decent size need adequately resourced, multidisciplinary data and analysis teams!)

In general though, I’d say a key facet of most data insight roles is to be a bridge between the technical detail or statistical nuance and the wider organisational context and objectives. People who can translate numbers, stats, or code into easily understandable findings and recommendations are worth their weight in gold.

I would also say that people who can simplify to the extreme (without dumbing things down) have a crucial role to play in widening the impact of data science in the social impact sector. So many organisations try to do too much. In the charity sector especially, the range of operations and systems in play can be staggering. Anecdotally, it seems the smaller the organisation the more complex and multidisciplinary their work often is. The inherent difficulty of all the data engineering and wrangling keeps it the reserve of technical specialists or expensive agencies.

A focus on getting solid basics in place – that are understandable and repeatable – would be very wise for most non-profit organisations.

What tools and methods can be applied to data science for social good?

There are so many tools and platforms out there now, it’s scary (with new ones every day – this will only accelerate with generative AI).

My general outlook is to be agnostic to any specific tool or technology. When you focus on how these systems fundamentally work (a bit of software engineering knowledge really helps here), you start to see all the similarities between tools and understand that many of them are just variations of the same thing. It also helps you to pick up new tools with ease, which is simply essential for the modern data scientist. The best analysts and data scientists can use whichever tools or techniques would suit the problem at hand.

That said, of course there are some very common tools which are standing the test of time and practitioners like myself often turn to. Here’s a basic shortlist of (perhaps unsurprising) tools to engage with:

• General purpose data exploration & wrangling; quick & dirty calculations: Excel (it just won’t go away!)
• Statistics, modelling, machine learning, and data visualisation: Python and R
• Statistics & modelling: PSPP (free version of SPSS, originally standing for “Statistical Package for the Social Sciences”)
• Database querying & management: SQL
• Data visualisation & dashboards: Power BI and Looker Studio
• Web analytics: Google Analytics, Piwik, Hotjar, Microsoft Clarity, and Convert

When it comes to techniques, staples for me have been regression analysis and applying significance testing to randomised controlled tests. This will vary hugely depending on your expertise and the business context you work in. The above techniques are not super advanced, but applicable to a wide range of descriptive and predictive analytics.

And again, to reiterate the point about getting the basics right: just following a rigorous analysis process (validating the data at every turn, using appropriate comparisons and distributions, thinking through questions and context, avoiding taking results at face value) will get you very far.

What sort of education or training makes for a good data scientist or analyst?

Everyone will have their own path and I think it’s really important to encourage diversity in the sector. A technical background or direct training in data science or stats will help, but to be honest in the world of data I have seen people with a hugely diverse range of backgrounds (technical and non-technical) be successful. You could produce the best, most accurate analysis in the world, or the prettiest report, but if you don’t appreciate the context or are unable to communicate to the people that matter, it’s a bit of a failure.

I would suggest to any budding data scientists or analysts to follow your own interests and what you enjoy, while being able to make positive impact.

Remember also that data can be qualitative or quantitative, and these days there is a big focus on combining the two: something I’m personally very passionate about. This perhaps opens up data analytics and insight to a large cohort of people who wouldn’t have traditionally considered themselves to be “good with numbers”. (Oh, and by the way: quantitative data analysis is about patterns, not numbers!)

There are also countless great online resources for learning all about data science – many of them free. You just need to commit and put the quality time in. Personally, I find that EdX and Coursera have many brilliant courses in data (as well as the social sciences).

Do you have any additional guidance for those looking at a career in data science for social good?

My first point would be: just go for it!

Whether you are just starting out or making a later career change, get stuck in with real problems, datasets, and tools. Find what makes you tick. There are so many areas even within the data science/analytics niche that, regardless of your skill set or interests, you are likely to be able to make a good contribution.

The “data” field is now so broad that I would recommend focusing on one core discipline at first – e.g. data visualisation, data science, data engineering, predictive analytics – and becoming expert in that.

Having real examples of how you have put your skills into action – and what impact they had on a project or organisation – is also invaluable.

The beauty of modern times is that there is already a lot of social impact data out there to work with, for example the World Bank Open Data, data.world, Kaggle, or even data published by public bodies like the UK government.

Always aim to be be collaborative, open, inquisitive, and engage teams on their own terms.

It’s really tempting to jump straight into a solution or nice looking report, but it’s so important to understand the needs of the team, organisation, and sector first. Then you can be much more focused and effective in your work. Patience and persistence really are key to both data science/analytics and working in the non-profit world.

Data science has huge potential to make a significant impact in the social sector. By using data to inform decision-making and improve outcomes, we can help create a better world for everyone.

Peace.