Thank you for the opportunity to comment on the enduring role of the National Archives of Australia. I first made use of the Archives as a history honours student in 1984, and since then have held a deep appreciation of the richness and significance of the Archives collections. I was employed by the Archives from 2007 to 2009, working on the development of a number of online projects such as 'Mapping Our Anzacs'. Since that time, I have pursued research in the digital humanities, often using the Archives collections to demonstrate what is possible using digital research methodologies. An example of this is provided by my keynote address to the Australian Society of Archivists Annual Conference in 2016.1
I love the Archives collections. They are critical both to democratic accountability and to our understanding of Australian history, politics, and culture. The Archives preserves not only the workings of the Commonwealth government, but the interactions of the government with ordinary individuals. These perspectives are not captured anywhere else and provide an import counterbalance to other historical sources. An example of this is the research I have been undertaking with Dr Kate Bagnall on records relating to the administration of the White Australia Policy.2
But like other users of the Archives, I am frustrated by limitations around access and a lack of transparency around the Archives' policies and practices. It seems, at times, that the Archives regards researchers as a burden to be managed, rather than a core constituency and a source of useful collaborations.
For a number of years, I have been harvesting data from RecordSearch relating to files with the access status of 'closed'. My purpose in doing this is to investigate the reasons why files are withheld from public access. The Archives provides summary data relating to access decisions in its annual report (though this information was absent in 2016-17). However, more detailed information documenting which files are withheld and why is not made directly available. Harvesting data from RecordSearch is difficult and prone to error, but it is currently the only way in which such questions can be examined. I reported on some of the issues arising from this data in an article for Inside Story in 2018.3
The Archives rightly notes that the overwhelming majority of files undergoing access examination are made available to the public with minimal delay. This is, of course, just as it should be. Under the Archives Act, records more than twenty years old are expected to opened to the public unless particular exemptions apply. But the data I've harvested shows that the number of files categorised as 'WPA' (Withheld Pending Advice) has been steadily growing. The summary data released by the Archives does not reflect the cumulative impact of this trend – the fact that a growing number of files have been waiting for years for an access decision. Moreover, it does not identify specific bottlenecks. The harvested data shows, for example, that files referred to DFAT for advice can be held up for long periods. Most historians working on modern Australia have horror stories about files that took years to move through the access examination process. PhD students are warned away from topics relating to foreign policy that are likely to be hamstrung by delays in access examination. Clearly this impacts not only on the research value of the Archives, but its public reputation.
The access examination process was reviewed by Paul O'Sullivan (formerly a deputy secretary in DFAT and head of ASIO) in 2016, but the results of this review do not seem to have been made publicly available. Once consequence of this review, however, seems to have been changes to the Archives Act in 2018. These changes largely seem to have been designed to discourage researchers from making more than 25 access requests. These changes reflect both the lack of transparency around the examination process (what did the review actually recommend?), as well the tendency to blame researchers for systemic problems in the access examination system.
While I would strongly argue for increased resources to enable the Archives to do its job effectively, I also believe that improvements in the access examination process require changes in governance, policy, and legislation. There seem to be two major issues. The first is that there are no public guidelines around when or why files are referred back to agencies for advice. Section 35 of the Archives Act says that the Archives can do whatever it thinks is necessary to determine the exemption status of a file, but there is no explicit mention of the role of agencies in this process. The Archives has produced a detailed and public policy document concerning exemptions 33(1)(g) and 33(1)(j) that relate to privacy and business affairs. A similar set of publicly-available guidelines that spelt out how the national security exemptions are interpreted and applied would at least help users to understand why files are 'Withheld Pending Advice' and shine some light on the decision making process. It also seems anomalous that while the 'open' period is defined by legislation, 'closed' files remain so indefinitely – until another access request starts the examination process all over again. Just as the policy relating to 33(1)(g) and 33(1)(j) sets some guidelines around the periods for which such exemptions should apply, surely it is possible to define maximum time periods for which national security exemptions should be valid. This would help streamline examination processes and give researchers some clarity around access. It is also worth noting that if the Archives released more data relating to its access examination process and outcomes, this could encourage collaborations with digital researchers who could help to identify risks, trends and bottlenecks. The work of Columbia University's History Lab is a good example here.4
The second issue is how government agencies themselves manage requests for advice they receive from the Archives. As noted, the role of agencies in this process is not mentioned in the Act, so agencies have no specific obligations to respond within a particular timeframe. I believe this part of the Act needs revision to impose clear responsibilities on agencies. There might, for example, be a time limit for responses from agencies— if an agency does not respond to the Archives within a specified time frame, then it is assumed they have no objection to the file being released. This would encourage agencies to improve efficiency, focus resources, and prioritise their decision making.
But this raises another problem with the Archives' functions and governance arrangements. For the Archives to fulfil its role to preserve the records of government, it needs to develop and maintain close working relationships with other government agencies. But this might make it difficult for the Archives to push back against agencies who are slow to respond to access requests, or who apply overly conservative interpretations of the exemption categories. What is missing is an independent advocate for public access – a body that can shield the Archives from having to police the behaviour of agencies.
It is interesting that in the UK the Advisory Council on National Records and Archives is independent of the National Archives and 'represents the public interest in deciding what records should be open or closed'. Included in its role and principles is the statement: 'The Advisory Council regularly challenges government departments to provide evidence to justify requests for permission to retain documents or for them to remain closed'. I do not know how this works in practice, but in principle it seems that the Advisory Council helps to keep the agencies in line, allowing the National Archives itself to focus on the collaborative aspects of the job. The Advisory Council of the National Archives of Australia has a much more limited role, indeed it is not clear what they actually do. I believe Australia should have a formal body that represents the public interest in questions about whether files should be withheld and is prepared to argue back against agencies as required.
There are other ways that the access examination process might be improved. As the data in my Inside Story article (and at http://closedaccess.herokuapp.com/) suggests, a number of the exemptions under the Act are rarely used. In particular, there seems to be an argument for combining a number of the national security exemptions. This might not speed the process up, but it might make it easier to understand. On the other hand, there is currently no exemption for culturally sensitive material relating to Indigenous Australians. It seems at the moment there is a policy of restricting such material by using the ‘privacy’ exemption, but I think an explicit category would be more appropriate. Perhaps it’s time for a complete review of the exemption categories?
The other side of public access is how the Archives collections are delivered for use. The fact that RecordSearch, as a collection interface, is well past its 'use by’ date should not distract us from the fact that the data model that underlies RecordSearch, the Series System, is rich and full of possibilities for research and exploration. However, if this data is to maintain its value, increased investment is required in archival description and metadata management. This fundamental work is crucial in facilitating long term access.
The Archives has spent considerable amounts of time and money over the years in exploring alternatives to RecordSearch. It is not clear what has been learned through these projects, and what the priorities of the Archives are. At the same time, the Archives is rightly investing much time and effort into moving agencies to a digital only framework. Indeed the Archives has been collecting born digital records for many years now. But where are they? It seems likely that some of the born digital records are now in the open period, but, as far as I know, there is no system for requesting or gaining access to these records. I believe the Archives should develop and release a set of principles guiding the design and development of its online systems. This would encourage further debate around the nature of online access, and could lead to the development of collaborative approaches across the GLAM and academic sectors.
In particular, I believe the Archives should commit to making more of its collection data available in machine-readable form. Some data dumps have been published through data.gov.au, but these do not seem to be maintained. There was discussion of the development of a collection API some years back, but this seems to have come to nothing. Lacking other options, I have developed a series of tools for 'screen scraping' data from RecordSearch.5 Screen scraping is inefficient and prone to failure, but it has enabled me to undertake a number of projects that investigate and enrich the Archives collections. For example, 'The Real Face of White Australia' encourages volunteer transcription of records from the Archives, and makes the derived data publicly available for further research.6 By making its collection data freely available and openly licensed, the Archives could encourage innovative research in a number of fields.
When the Archives implemented its ‘digitisation on demand’ scheme, the idea that individual users would pay for digitisation but the item itself would be made publicly available was quite innovative. This approach has meant that digitisation has followed a broad range of research interests. I suspect, however, that the huge increases in digitisation fees in recent years has essentially killed digitisation on demand. It is now often cheaper to travel to the Archives and photograph what you want, so the broader public benefit of digitisation is lost and we all have our own private collections. With continuing cuts to its funding over a number of years, it is understandable that the Archives would seek opportunities for cost recovery. But I wonder if the broader public benefit of the ‘digitisation on demand’ scheme was factored into the costings. Again, more transparency around such changes would be valuable. What is lost, as well as what is gained?
The other side to the Archives digitisation program is ‘pro-active digitisation’ based on the Archives own assessment of priorities. I asked a couple of years ago if there were any publicly available guidelines around the assessment of priorities for pro-active digitisation, I have yet to receive a response. Again the lack of transparency around such things is disturbing. I have tried to assemble some data on how digitisation shapes the Archives online presence, but it is difficult to to analyse how the shift away from digitisation on demand will change the overall picture.7 Given the significance of digitisation in facilitating online access, what is the Archives long-term strategy?
The National Archives of Australia, like other national collecting institutions, has suffered from ongoing cuts to its funding. This needs to stop. Our cultural heritage is being threatened by short-term thinking and political games.
But while I am strongly in favour of increased funding for the Archives, I believe that the Archives could do much more to increase the transparency of its policies and decision making, to strengthen its relationship with the research sector, and to open its collections to new and innovative uses.
Dr Tim Sherratt
timsherratt.org
Footnotes
-
‘Turning the inside out’, http://discontents.com.au/turning-the-inside-out/ ↩
-
Tim Sherratt and Kate Bagnall, ‘The people inside’, in Kevin Kee and Timothy Compeau (eds), Seeing the Past: Experiments with Augmented Reality and Computer Vision for History, University of Michigan Press, 2019, https://doi.org/10.3998/mpub.9964786 ↩
-
Tim Sherratt, ’Withheld pending advice’, Inside Story, February 2018, http://insidestory.org.au/withheld-pending-advice/ ↩
-
GLAM Workbench, https://glam-workbench.github.io/ ↩
-
Transcription site: http://transcribe.realfaceofwhiteaustralia.net/; data repository: https://github.com/wragge/realface-data ↩
-
http://timsherratt.org/research-notebook/aggregated-archives/notes/naa-functions-view/ ↩