Interview with Professor Sir Ian Diamond

Sir Ian Diamond, was born in March 1954 in south Devon. His family was close and he and his brother enjoyed a happy childhood with education at it’s heart. Neither of his parents had been highly educated but had a great regard, he says: “They did everything they could to enable education for my brother and myself.”

The family lived just outside of the holiday town of Torquay. Ian says: “It was thriving with many opportunities for employment, be they in the summer holiday or any other time, and much more so than the coastal towns of today where the move towards international travel, pre-pandemic, has impacted pretty negatively on the wellbeing of the populations of coastal towns.” Ian took advantage of the opportunity to work during his holidays all through his university holidays, with jobs as a waiter, teaching English to foreign students, and in refuse disposal.

Having passed his eleven-plus, Ian attended Torquay Boys’ Grammar School. He says of the experience: “As with most people, it was a mixed experience. I thought the school was great.  I did not always work hard, but I inspired by some very good teachers, particularly in mathematics, where I enjoyed both the teaching and the subject very much. I also enjoyed history very much, and it was a history teacher who gave me some extremely good advice that if I was going to go to university I should go to the London School of Economics, and that was one of the best decisions that I made, and that advice is something that I’ve always remembered.”

 Of his A Level choices, Ian says: “I’ve no idea why, but I had always had an interest in the social sciences, and in particular in the way in which surveys helped us to understand populations. It was something I’d always been interested in, although I could not for the life of me tell you why that should have been the case, and that led me to do A levels in maths, further maths and economics.”

After gaining reasonable A levels and with the advice of his history teacher, Ian went to the London School of Economics to study for a BSc Econ. The course had several sub-categories allowing students to experience and then select the elements that best suited their interests. Ian adds: “While it was a BSc Econ, you spent most of your time doing whatever was your specialisation. During one’s first year there was a requirement to take some economics. One of the great things I also enjoyed was during one’s second and third year there was a requirement to take some courses from outside of statistics, but the great majority of the work I did at London School of Economics was statistics.”

 He says of his interest in statistics, “I always found it really super-interesting and maybe sometimes not knowing that it was statistics, but numbers excited me and understanding numbers excited me, whether it was a league table or whether it was just a cross-validation. I had always found that fascinating and that led me to doing statistics and perhaps without knowing too much at that time about what that really entailed.  I think information for potential students is much better now than it was then, but certainly I enjoyed every moment of my undergraduate career.”

Ian remained at LSE to complete an MSc to further his knowledge, he explains: “I stayed at LSE for the very simple reason that looking at other Masters courses, LSE was the closest one to where my interests of applying statistics to the social sciences and being able to do some courses in demography, social surveys, social statistics.”

Not sure what to do after his Masters, one of his lecturers suggested doing his PhD. As Ian was considering this, he read a paper by Richard Cormack in the journal of the Royal Statistical Society, he adds: “I thought it was completely brilliant and entirely serendipitously, about a week later, a postcard from Richard Cormack appeared on the departmental noticeboard advertising a PhD grant that he had got through the Economic and Social Research Council (then called the Social Science Research Council) inviting applications.  I took a view that I’d never been to Scotland in any serious way and a number of people said to me that Richard Cormack was a very nice man, and so I applied. I was incredibly lucky to get that scholarship, and off I went to St Andrews and wrote a PhD.”

The experience allowed Ian to work on a real problem looking at why larger numbers of Scottish students who had taken Highers dropped out of their degree courses at St Andrews than English students. He adds: “I addressed a whole series of questions, some of which were substantive, inasmuch as why was that, and others which were statistical, which were if you’re going to predict educational performance at university based on school performance, then how do you put numbers on grades at Higher, A, B, C, do we put that three, two, one; four two one, and A level in a way where the grades are comparable. It’s comparable to have an A level ranking with numbers on and a Higher ranking on numbers, the two are comparable.  It’s an interesting problem and one that I enjoyed working on very much. I’m not going to say the answer, I got an answer.  Was it the perfect answer?  Probably not, but it worked for those data and I think it helped people to understand that the importance, particularly for Highers of the A grade, as opposed to B grades and C grades, and the extra weighting that one needed to give.”

 In working on the problem, Ian worked with computers. He says: “Some of the programmes I wrote required an enormous amount of computing time and the only way that could happen was actually to send the punch cards, great boxes of them, to Cambridge where they were ran on a big mainframe there and where one would get the answers back two or three days later. One had to really be careful that the punch cards were in the right order and that the programme had been written well.”

Ian also had access to another mainframe at St Andrews which didn’t require punch cards but used a screen. The university had also set up a computer with Heriot-Watt University and two others and St Andrews students could use their data. This became helpful to Ian who took a one year lectureship teaching at Heriot-Watt university while he completed his PhD. He adds: “It was fantastic because I didn’t have to move everything and learn a new computer skill, I just basically never changed my password.  At that time that was totally path breaking for me, it was so exciting that I could be in a different city looking at a screen and access my data  and that enabled me very much to finish my PhD very successfully.” Having handed in his PhD in the spring of 1980, Ian sat his oral exam later that year with one of his heroes in the field of statistics; Harvey Goldstein.

During his one year lectureship at Heriot-Watt, Ian saw a job advert for a role at Southampton. He explains: “Southampton at that time, alongside London School of Economics, were the two places that really were doing social statistics. It was a lectureship majoring in demography, which was something I’d taken in my undergraduate time, and it was in the Department of Social Statistics. I was incredibly excited, I didn’t think there was a chance that I’d get it.” Ian joined the small department led by Tim Holt, on 1^st May 1980.

He adds: “The department had five lecturers and two secretaries. It was a place of great energy, of great enthusiasm and a place where you felt you could just get on and do things and you would be supported. And I was supported, not only there, but also initially Tim Holt had arranged for me to spend a day a week at the Royal Fertility Survey, which was a big international survey based in London doing surveys in low-income countries across the world. That was a great opportunity to be able to interact with colleagues from across the world and also to be able to analyse some incredibly innovative data and produce work on population, which was based on really quite innovative surveys at that time.”

 Ian taught the first and final year of the undergraduate programme in demography as well a number of subjects on the Masters programme in social statistics.

 He says: “I love teaching. It was a great responsibility to produce good research, and I got involved in a number of good projects and I really enjoyed those, but at the end of the day the teaching was critical to what you do. I take a view that if you want ever to put ‘Professor’ in front of your name, and I didn’t at that time, then you have to be prepared to profess your subject, and there is no better place to profess that subject than to first year undergraduates.”

Ian remained at Southampton until 2002. He adds: “I had opportunities to go elsewhere, but you simply had to ask yourself what would I get by moving. We built up a department, we got great PhD students, we brought in some more lecturers because we had new programmes setting up, we had some great research projects.  We had a big one once from the ESRC which was really broad.  It was fantastic working alongside people like Chris Skinner who is just a brilliant survey statistician, as well as John McDonald with whom I published a whole series of things with Phil Cooper. It was just a great place to be and you felt things were exciting, you felt things were moving forward and how could you achieve anything better than you were achieving, so stay and get on with it.”

Ian was asked to become the Dean of Social Sciences. He says of the experience: “I enjoyed that. I learnt an enormous amount about leadership and across a wider range of disciplines.  Then, I was asked if I’d put my name forward and be interviewed for the position of Deputy Vice-Chancellor. It was a much fiercer interview than I was expecting, but I was successful and I enjoyed that role very much as well.”

 The departments first computers were adopted in 1985, however, as they were not powerful enough to run the necessary programmes quickly, Ian continued to use the university’s mainframe. He says: “We were analysing some pretty big datasets. Not enormous datasets compared with the sorts of things one can do now, but they certainly seemed to be pretty big in those days. Some of the computational programming we were using was working at the extent of the space that even the mainframe had and so it really wasn’t an option at that time to do major analyses on anything other than the mainframe.”

In 2003 Ian left Southampton University and moved to ESRC as Chief Executive Officer, having previously served on the grant awarding boards and council. He says: “I enjoyed it and felt there were things that I might like to do were I to be the chief executive, when the then chief executive, who I thought was doing brilliantly, moved to a new role. So when Gordon Marshall went to be Vice-Chancellor of the University of Reading, I had a conversation with one or two people who encouraged me strongly to put my name forward, so I did. I did so in hope rather than expectation, and I had a really interesting interview, after which I was privileged to be invited to take the job.  It was one of the very few jobs that would have moved me from Southampton, because I saw it as a job which had a real opportunity to move social science into the mainstream of the research base, an area that I didn’t believe it had been at that time.  Secondly also to really further the needs for a modernisation of data availability in the social sciences and to be able to impact massively on policy and on people’s lives.”

Ian stayed at the ESRC for seven and half years just short of the maximum contact time of eight years. During those years, with the help of his colleagues, he says that they “were able really to drive forward a vision that social science was incredibly important, not only in and of itself, but also in partnership with many of the natural sciences to bring a true multidisciplinary approach to many seriously important problems. Those are the kind of things that we think of now as being obvious, but at the time some of that was actually quite innovative.”

 Ian gives one example of the change the team created when they persuaded David Rindt to chair a national data strategy. He explains: “That national data strategy gave us the case to really expand the sample size of what was then the British Household Panel Survey into what is now known as Understanding Society. It was a really brilliant panel study and with great help from John Hopcraft we were able to make the data collection include biological data as well as social data. We also enabled the development of a lot of disciplines and also brought ESRC into the heart of a number of initiatives.”

The council under Ian’s leadership also worked with various organisations including   Cancer Research UK, the British Heart Foundation, the Wellcome Trust and the Medical Research Council to build public health centres, with the Arts and Humanities Research Council to build centres which focussed on language learning. Ian adds; “We really did do some really interesting things and we brought social science much closer to both the rest of the research base, but also to policy.”

In 2010, after completing his tenure at ESRC, Ian was invited to put his name forward to lead the University of Aberdeen, where he would remain until 2018.

He says of the role: “It was varied.  I enjoyed a lot of it enormously, and together with my colleagues we were able to bring a greater focus on teaching and on the student experience, and that was reflected in 2018 by being awarded the Scottish University of the Year, something that I think was a great tribute to a lot of hard work from a lot of people.

As well as leading the University of Aberdeen, Ian was also involved with some external activities including working with Universities UK and Scottish Universities around efficiencies which lead to the formation of the group which designed the Higher Education Funding System for Wales.

After leaving the University of Aberdeen to, Ian took a year out to concentrate on some further education projects, including acting as chair at Edinburgh College. It was at this point that he was invited to put his name forward for the ONS as National Statistician.

He says: “I had always loved dearly the work of the ONS, I’d been privileged at the end of the 1990s, beginning of 2000s to work on designing an under-enumeration strategy for the 2001 Census, which was subsequently used both in 2011 and in 2021.  So it didn’t actually take too much persuasion to put my name forward.” Ian started part-time in August 2019, and moved to fully time in mid-October in 2019. He adds: “It’s been a total thrill every day since. It’s unbelievably exciting. At a moment in time, where increasingly people are aware of the need for accurate, timely data in order to make policy which impacts positively on the lives of every citizen in our country, we are charged with producing those data and producing them in a way which the public can have trust. That’s been an incredible opportunity.  I very often tell my colleagues that we need to recognise that our job is to reflect the economy and to reflect society and in so doing both those change very quickly. We’re an organisation in a permanent state of change, we always need to be changing what it is that we are doing because what we are trying to measure is changing.

“Then, if you bring along with that the massive opportunities that increases in technology give us, both in terms of being able to think incredibly radically about what data are.  We should continue to undertake surveys, we should continue to measure things as accurately as we can, but equally we can use all kinds of different sources of data, masses of different types of data, many of them born digitally, which can be used to produce ever more timely, ever more accurate statistics.

“At the same time we have also now got the technology to be able to merge datasets and that just gives you the opportunity to do things that you could only possibly have imagined in the past.”

As an example, Ian points to the worry during the pandemic of the way in which the Covid-19 virus is impacting disproportionately on people of different ethnic groups. He says: “What we needed to do was to take death certification, which doesn’t include ethnicity for very good reasons, link that to the Census data to get ethnicity, then link that to, for example, Valuation Office data to understand housing, so as to be able to have a handle on inequality, and also link it to health data in order to be able to get a handle on any comorbidities.  When you do that, you’re able to undertake some analysis which shows you quite clearly that over and above controlling for disadvantage, people from particular ethnic minority groups had higher levels of mortality, and then you can start to say quickly what is it that we need to do about that.”

“During the whole period since I’ve been at ONS we have innovated with new surveys, with new data collection, with looking very carefully at how to use multiple sources of data to address big problems, and I think we’ve been extremely good in doing that.”

On the image of statistics, he says: “Statistics has had a bad press over a long period of time.  I do actually think that rigorous, well put together statistics are incredibly reliable and are incredibly powerful.  There are now a number of people who would call themselves data journalists, at places like the Financial Times or at Sky News, for example, who are absolutely brilliant and for whom I would not say, under any circumstance, that they misinterpreted data. One of the things I’ve really noticed in working with a number of journalists over the last couple of years, is a real desire to understand what the data really say and to report them entirely accurately. That is something that we at ONS really need to think very carefully about. Our job is to provide the statistics, not only to provide the statistics but to communicate them in a way which is understandable to all the public, and to do so in a way which ensures that every citizen has a voice in our data.  If you think about that, then you move into a position where you are able better to understand that statistics are not fraught with danger, but they just require a rigorous and careful use and an understanding of the assumptions underlying their collection.”

As official statisticians we have a responsibility to be able to continue to build real clarity in our exposition and real transparency therefore it’s so incredibly important that when we produce statistics, we do not just produce them as a number, we also produce the metadata around it, which explains what the assumptions are, what the pros and cons of these data are, so that people can really understand them and we’re not overplaying them.

“The other thing that is so incredibly important is to remember that most of the time what we do in statistics is produce estimates. Estimates have what is often a measurable error around them. We need to be able to produce that degree of uncertainty because if you’re saying our best estimate is, for example, the number is twenty-seven, and that’s plus or minus a half, that’s very, very different to saying the best estimate is twenty-seven, plus or minus twenty-six. Really understanding what we understand by uncertainty is incredibly important.”

Asked about how artificial intelligence and machine learning is impacting statistical production, Ian says: “There are things that you can do, for example, with textual analysis that I just think are unbelievably exciting. Having done some textual analysis a long time ago with focus groups and knowing how difficult and hard it was to do, what you can do today with machine learning is fabulous.  There’s a lot of really exciting things you can do with machine learning, there is real potential, for example, in privacy enhancing technologies, and I am extremely enthusiastic in getting all the best things we possibly can from machine learning.  Having said that, I still believe that there is an important role for really understanding the problem and having a social theory underlying what it is that you’re analysing, and then your machine learning or your ever more powerful statistics can be able to iterate two solutions which really do inform the public much better than you could otherwise”.

Asked what might lie in wait in the field of social sciences, data science and statistics, Ian says: “If I step backwards ten years I would probably have said things around some of the data linkage that we are now doing, but I would not have said so much around the use of, for example, telephony data or scanner data, that’s just seemed so otherworldly.

“Over the next ten years, we will continue to be able to access data and to merge data and to address questions that one could only have imagined addressing when I was much younger.  The power to be able to analyse those data will be ever greater, but at the same time we need to recognise the threats that come from cybersecurity. Therefore we need to be doing everything we can to ensure the safety of data and to ensure that we have the right governance, the right public engagement, the right approvals, the right ethics, for our use of data. That will become ever more important in ensuring that we’re able to have the public permission to analyse some of the ever more complex datasets that we will be able to do using the kinds of machine learning, using the kinds of very powerful iterative statistics that we now have.

“For an official statistics institute, there are two things I would highlight. The first is that we have to take a greater role in producing statistics which are accessible and easy to use by the public, broadly defined. Secondly, we need to be ever more radical and ever more ambitious in the way that we produce statistics at pace and produce statistics that are timely, and produce statistics which give every citizen a voice and that we do so quickly and inspirationally and in a way which means that they are accessible.”

Of his proudest moments, Ian says: “Being knighted in 2013 was on a personal front, an enormous thrill and enormous source of humbling pride. However, I would also point to the work I did in Wales, to some of the statistics I’ve analysed, to the work that together we did on under-enumeration that stands up over the course of time, some of the work I did in the 1980s around student projections. These are the things I’m very, very proud of.  So while the knighthood is just the most incredibly wonderful thing and something that I’m incredibly proud of, I would rather think of some of the reasons why I got that knighthood as being contributing to, in a small way, to improving our understanding of society and improving in a small way some people’s lives.”

For anyone considering working in statistics and data science, Ian says: “It’s  unbelievably exciting.  For those people who are interested, there can be no better career, particularly if you really keep a focus on real world problems that require analysis and numbers; you can just move from one thrill to another.”

Interviewed by Jane Bird

Transcribed by Susan Nicholls

Abstracted by Lynda Feeley