Shadow Culture Minister Ed Vaizey MP, speech on Archives in the Digital Age

In 2006 we created another 161 exabytes of digital data – more than the sum of all data created up to that point.

YouTube now hosts 100 million video streams a day and London's 200 traffic surveillance cameras transmit 64 trillion bits a day to the data command centre.

In Government, a relatively small Department like Culture, Media & Sport were even ten years ago adding a volume of data equivalent to the Complete Works of Shakespeare to its computers every day.

Five years ago, the same volume was being added every hour.

Today, they are doing so every minute and by next year it is estimated that they'll be doing so every second.

Our very concept of archives and records has changed. The explosion of digital information means that new ways of keeping records, of archival practices and user interaction are being developed to meet the new digital challenge.

The challenge is simple but huge. In short, archivists need to find ways to ensure that born-digital records are useable in 25, 50, 100 or even 200 years time and that the records we have from the last 200 or more years are available to a wider audience via electronic means.

Digital continuity and digital preservation are now as essential to the framework of the archival profession as parchment conservation and traditional cataloguing.

There is a real challenge to the archive and record keeping profession, especially as resources both in the public and private sectors get squeezed in the coming years.

Whilst digital archiving and working with born-digital records can be spun as creating smarter working and leading to cost reductions, in the short-term there can often be front-loaded costs, such as with digitisation of paper documents, or the development stages of new archival software.

For many in the archive sector, the additional burdens placed by the digital agenda can seem like they are being asked to do more stuff with the same or less resource.

The National Archives in Kew has looked after Government records going back to the Domesday Book. One thousand years of our nation's official history, mostly in paper record form and telling the story of the development of our society.

These records have had to be catalogued, preserved and made accessible over a long period of time and there is an expectation for this to continue during our lifetimes and well into the future.

The same challenges exist for digital records, but they cannot be addressed in the same way as with traditional archival materials. Digital data cannot be file away like paper and for it to be assumed that it will still be readable and useable in 30 years time, because it won't be.

Digital data is not just more vulnerable to obsolescence than paper records, but also there is just so much more of it to keep track of, store, catalogue and make accessible. The digital revolution has created so many different types of record – from the basic document which my speech was written with, to the email it was sent to me and the website and blogs it may appear on and the Tweets which will be generated in an attempt to get people to read about what I have said here this morning.

And not to mention all the texts (or SMS) messages, wiki sites, and discussion forums and so on. There really is a huge question mark over what even constitutes a "record" in the digital age.

The growth in demand for online archives continues apace.

270,000,000 online documents are now accessible through the UK Government Web Archive. Ranging from the Magna Carter through to records of the National Coal Board.

1,680,551 free copies of records have been downloaded from The National Archives Documents Online service in 2008-09.

Yet even after all this, only 5% of government records are selected for preservation in the collections of The National Archives.

Expectations of growing content and availability of the rich holdings of archive repositories across the UK do not relent. Examples such as the letters of David Lloyd George, held at the National Library of Wales in Aberystwyth, being digitised and made available online is a fantastic way to widen access to an important collection beyond researchers with the resources to travel and study in West Wales.

Whilst traditionally access to the archival holdings in our local authority and national repositories have been free of charge to the user (at the point of delivery, at least, if one ignores the general taxation which has funded their development), the online market is different and where value and convenience are enhanced, users are willing to pay for certain content.

The huge popularity of the 1901 and 1911 Census online services shows what the market can sustain in terms of charging for access to archives. Local authority archivists are also reporting similar trends, with an increase in demand for online information there also comes a willingness to pay a charge for the service. This pay-as-you-go approach to using archives in itself provides an interesting challenge to the traditional working methods of the archivist and changes the relationship between user and provider of archival services.

The recently published Government Archives Policy "Archives for the 21^st Century" identifies a number of key themes which seek to address the digital challenge facing the archive sector. The policy recognises how important it is for the archive sector to work collaboratively with other sectors, including higher education, local authorities and the private sector to enable the delivery of more comprehensive and better quality services to the user.

The growth in the networks and systems to share digital information securely across sectors has led to a rethink as to how services are offered to the user. From the user perspective is matters less as to where the information originated as opposed to how easy it is to access and use. This new way of thinking is leading to a situation where it is Digital Partnerships which are breaking down the old Analogue Silos.

Archive services across the UK are also facing the challenge of how to provide excellent service standards with stand still or decreasing budgets, a situation which will no doubt be exacerbated by the expected continued pressure on public spending over the coming years.

The sustainability of smaller local authorities offering archive services with only one or two qualified practitioners is under question in the "Archives for the 21^st Century" policy document. The case for fewer, bigger and better services can be made across many areas of public service delivery, but in archives there are sensitivities related to geography, local history, culture and political concerns.

The digital revolution can provide many opportunities for local authorities to deliver archive services which benefit from the efficiencies and savings of centralised service organisation, whilst also taking into account local needs.

The UK has, uniquely amongst the nations of the world, a fantastic opportunity for leadership in the area of partnerships between the cultural and archival worlds. Already world leaders in the way we collect, preserve and make accessible our nation's rich and diverse history, the United Kingdom also contains institutions showing the way in terms of new approaches to archives.

To pick just three of our largest players in this field – The National Archives, the British Library and the BBC – we will all already have some ideas as to how these bodies use and present their archival holdings.

The BBC news site and educational website; The National Archives providing a gateway to the nation's official history and the British Library's recently launched online newspaper archive are all examples of how archives have been used in creative and user-oriented ways.

The perspective of the user is also highly important when it comes to considering the access issues connected with digital archives. A digitally literate audience has expectations in terms of usability, access and interactivity, as well as value and quality of provision.

Web 2.0 technologies have opened up different ways for information to be shared online and the formats in which user participation can be enabled.

The National Archives stand alone website "Your Archive" allows for users to make their contribution to the record. The site provides a forum where people can share their knowledge of British history and archival sources held by The National Archives, and by other archives throughout the UK.

Whilst these initiatives are all to be welcomed and have undoubtedly improved access and quality, they continue to be created and developed by the organisation itself and do not, as yet, add up to a coordinated attempt to marry together archival holdings, new technologies and the wider cultural sector.

The challenge for government in this area over the coming years will be to provide the environment within which a more coordinated approach to delivering user-oriented digital services in partnership across the whole range of cultural domains.

The United Kingdom's strengths in this area include our long history of developing traditional archiving methods and our long established network of national and local archive services. Countries from the EU-bloc, Africa and beyond often use the UK as a model for the development of their own networks and institutions.

The UK benefits from English remaining, to date, as the global language of commerce and culture. Coupled with the 1,000 years and more of recorded history in our archives, the very records of our nation serve as a reminder of our role and responsibilities in developing access and re-use of this kind of data.

What we need now is the leadership to take full advantage of the wealth of assets built up by our nation's history and to provide the vision within which our cultural sector can work with our archivists to create an even more enviable record of digital success which will last well into the next several decades.

The impact of this work will not just be to offer users the types of services and delivery options they are demanding, but the wider impact on society will be immense.

The Big Society idea, which draws a big distinction separate from ideas of the big government or the big state, shows us that communities are extremely interested in the records and archives that tell a story relevant to them.

The growth in online family and local history research projects and the opportunities for non-geographic communities of interest and shared heritage to exchange knowledge make digital archives even more relevant to a bigger agenda.

Archives play a huge role in promoting a sense of place, a sense of belonging and a sense of national, local and community history.

I remember when I went to The National Archives and was presented with the 1911 Census return, written by my grandfather
who was a barge builder on the River Thames – at Vaizey Wharf indeed – I had my own Who Do You Think You Are moment…the spine tingling sensation of coming into direct contact with one's own family history.

I know how important this personal interaction with the archives is and will do all I can to ensure that as we move into more digitally-based ways of providing these services, the unique user experience is not lost.

It is clear that the nature of generating records continues to change at breakneck speed and this poses a huge challenge to how professional archivists develop and manage archive services that remain relevant and accessible to a wide range of user groups.

At the same time, the opportunities for partnerships within the archive sector in terms of delivering better, more efficient services seem obvious; whilst the potential for partnerships with the cultural sector have barely been scratched beneath the surface.