Clever Recordkeeping Metadata Project
The Clever Recordkeeping Metadata Project (CRKM Project) was established as a joint research project involving the Records Continuum Research Group at Monash University, the National Archives of Australia (NAA), the State Records Authority of New South Wales (SR NSW) and the Australian Society of Archivist’s Committee on Descriptive Standards (ASA CDS).
The aim was to bring together researchers and practitioners to investigate how standards-compliant metadata could be created once in particular application environments, then used many times to meet a range of business and recordkeeping purposes. The project wished to explore how to move away from the current resource intensive process of manual metadata attribution and stand-alone systems, towards an integrated suite of business systems and processes supporting recordkeeping functions.
The project received funding under the Australian Research Council (ARC) Linkage Grant scheme and ran from mid 2003 to 2006.
About the Clever Recordkeeping Metadata project
Metadata - structured context-rich information about business processes, agents, systems and information resources - is a vital tool in managing business transactions and related information objects in complex intranet/internet environments to support eBusiness and eGovernment.
In recent years, a suite of standards, best practice models and guidelines has been developed to address the challenge of managing electronic records and other information objects. The main objectives have been to support reliable, trustworthy and accountable business processes, and to provide better access to information resources online, particularly in the areas of eGovernment and eBusiness. Metadata guidelines or standards are now in place in most jurisdictions in Australia.
Implementation of metadata standards is proving to be problematic. The challenge is to automate metadata creation and enable sharing of business critical metadata between the business systems and applications that control workflow; manage knowledge; and create, manage and retrieve documents, records and the content of web sites. Metadata generation and deployment in such systems are currently resource intensive and application specific. Metadata creation is not usually fully automated. Metadata created in one application of potential relevance to other applications is not shared between applications. Auditor-General inquiries in various Australian jurisdictions confirm that in practice there are major problems with implementation of the standards1.
Metadata standards development envisaged implementation in integrated systems environments and the deployment of metatools that would enable the clever use of metadata. In practice, many of the types of contextual information (metadata) created and used by recordkeeping systems are also created and used in a variety of other business application environments, such as desktop document authoring applications, web content management systems, human resource management systems, work flow systems and archival control systems. But recordkeeping systems as currently implemented do not draw on these systems as sources of metadata; rather they re-create it – often in manual and resource intensive ways. A parallel situation exists in relation to resource discovery metadata. For example AGLS metadata is most often created retrospectively at the time a document is made available on a web site, rather than being automatically supplied by the software in which the document was originally created, although this application would have also created almost identical metadata in order to manage and retrieve the document.
Although data modeling, mark up language and syntax initiatives are addressing the data representation requirements for metadata translation and exchange, this functionality has not as yet been utilised in the systems that support eGovernment and eBusiness processes and recordkeeping. Moreover there has been little progress in relation to developing strategies and metatools for the translation of metadata values between schemas in these environments.
The Clever Recordkeeping Metadata Project aimed to develop a prototype to demonstrate how to overcome major barriers to the implementation of recordkeeping and resource discovery metadata standards, particularly in eGovernment. It was intended to provide an implementation model for the clever use of metadata in quality recordkeeping systems that capture and manage information that can support the reliability, authenticity, accessibility and usability of evidence of government decisions and activities for as long as that evidence is required. The development of the prototype and the implementation model would involve the prototyping and testing of innovative metatools that will enable the translation of metadata between different schemas, including a mini metadata registry.
Quality assurance mechanisms will also be developed during the prototyping as a critical feature of the prototype and implementation model will be the demonstration of the business utility of metadata, as well as the cost effectiveness and feasibility of having integrated systems and applications that permit metadata to be created and captured once and reused for multiple business critical purposes.
The conceptual framework for the project is provided by records continuum theory and the records continuum model, which support the development of complex, integrated systems and processes to manage records and archives in and through time, and across space.
The research outcomes of the previous research project, Archival Metadata Standards for Managing and Accessing Information Resources in Networked Environments over time for Government, Social and Cultural Purposes, Chief Investigators Sue McKemmish, Anne Pederson, Australian Research Council SPIRT Grant (commonly referenced as the SPIRT project) were taken as a starting point for this project.
Anticipated Outcomes and Benefits
The outcomes and benefits of the Project at the time of the research proposal were that the Project would:-
- deliver a proof of concept prototype supported by metatools, capable of demonstrating the business utility of metadata and the cost effectiveness and feasibility of having integrated systems and applications that permit metadata to be created and captured once and reused for multiple business critical purposes (addressed by project outcome areas 3, 4, 5 and 6)
- provide a model for new software and systems solutions to assist organisations to comply with standards in cost-effective ways that maximise corporate return on investment (addressed by project outcome areas 5 and 6)
- provide prototypes of new metatools and demonstrate how a metadata registry can enable metadata translation between schemas (addressed by project outcome areas 2, 4, 5 and 6)
- enable Australian business system developers operating in the resource discovery, document management and records management industry sector to better position themselves in the emerging international market in this field (addressed by project outcome areas 5 and 6)
- support compliance with recordkeeping and resource discovery standards designed to ensure quality records and archives, accessibility of government information, and the accountability of government (addressed by project outcome areas 1, 2 and 3)
The research methodology was designed within an action-research framework where a close alignment between the practical development of tools and active reflection on each stage of their development iteratively informs both the further development of the tools and also identifies challenges and issues to be addressed in an ongoing fashion.
The research involved the initial development of a proof-of-concept prototype to demonstrate that metadata re-use is possible and illustrate the business utility of recordkeeping metadata. From that initial proof of concept, the project intended to develop a more robust demonstrator available for wider dissemination.
More details of the first and second iteration of the demonstrator are available on the Research Method and Findings page.
- Australian National Audit Office, Recordkeeping (Audit Report No. 45, 2001-2002), 1 May 2002, and Recordkeeping in Large Commonwealth Agencies (Audit Report No. 7, 2003-2004), 24 September 2003.
The CRKM Project concluded in December 2006. The final report of the project details the agile, iterative, rapid prototyping techniques employed in the development of a proof-of-concept Metadata Broker featuring recordkeeping metadata translation and registry services functionality. The prototyping process enabled exploration of issues regarding implementation environments and infrastructures to support recordkeeping metadata re-use and interoperability. The related PhD thesis, Building Capacities for Recordkeeping Metadata Interoperability, explores these requirements for recordkeeping metadata interoperability in depth.
The results of the research reveal how the limitations of current recordkeeping standards and practices that are predominantly still operating within the paper paradigm inhibit realisation of the full potential of recordkeeping metadata as a business tool. Recordkeeping practices and recordkeeping metadata schemas are yet to be fully transformed to meet the requirements of the emerging technical environments.
The Project demonstrates the immediate utility, but limited functionality, of essentially hard-wired metadata translation services to current recordkeeping problems. It also illustrates the potentially much greater utility of recordkeeping metadata registry and brokerage services in emerging service oriented architectures. The lack of maturity of this emerging environment offers huge challenges, but also an opportunity for the recordkeeping community and academic contributors to be proactive in defining recordkeeping services, and the role of recordkeeping itself in this emerging environment in the short to medium term.
Final report of the CRKM Project (Word document)
Broad Research Findings
- There is currently a limited capacity to support recordkeeping metadata re-use because recordkeeping processes, practices, standards and infrastructure still largely operate in paper-based paradigms.
- Current recordkeeping metadata standards lack semantic precision, and canonical machine processable encodings, both of which inhibit their uptake.
- Deployment of a metadata broker modelled on the CRKM Metadata Broker I will provide immediate practical use in environments where recordkeeping metadata is to be passed from one known environment to another, for example between a specific business system and a records system.
- However the CRKM Metadata Broker I model is of limited flexibility, scaleability and robustness, as it relies on hand crafted metadata crosswalks.
- The notion of Metadata Registries as currently in use in the literature is a compound concept which needs to be broken down into more granular statements of functionality which include different uses of the term 'metadata registry' to include authoritative schema registries, and multiple functionality required to support specific services.
- The emergence of services oriented architectures and the re-articulation of metadata broker components as services in these kind of environments, as per the CRKM Metadata Broker II model, has the potential to provide greater return on investment and cost effectiveness.
- The services oriented environment is much hyped, and represents a significant shift in the technological environment, but the reality of sustainable implementation is likely to take many years.
- The recordkeeping profession has an opportunity with this anticipated lead time to both re-envision recordkeeping functions and processes in service oriented terms and to address and influence the broader technology community on the role of recordkeeping in the service oriented environment.
- Impact of different information paradigms on metadata interoperability
- Recordkeeping metadata standards
- Paper paradigms prevail
- Re-conceptualising the delivery of recordkeeping services
- Service orientation for recordkeeping
- Metadata registry functionality and general robustness of technology infrastructure
People and Partners
|Chief Investigator||Emeritus Professor Sue McKemmish, Monash University|
|Partner Investigators||Professor Anne Gilliland-Swetland, UCLA|
Mr Adrian Cunningham, National Archives of Australia
|Research Associate||Ms Karuna Bhoday (October 2003-December 2005)|
Ms Barbara Reed (January – June 2006)
|APA(I) Award||Ms Joanne Evans|
|Programmer||Dr Sergio Viademonte (August 2004-July 2005)|
Mr Andi Sumartono (August 2005-June 2006)
|National Archives of Australia Representative||Mr Duncan Jamieson (November 2003- August 2005)|
Ms Karen Skelton (September 2005-June 2006)
|State Records New South Wales Representative||Mr Tony Leviston|
|Australian Society of Archivists Committee on Descriptive Standards Representative||Ms Barbara Reed|
|Advisory Group Members||Chair: Mr Adrian Cunningham, National Archives of Australia|
Ms Kathryn Dan, Records and Archives Services, Monash University
Mr John Roberts, National Archives of New Zealand
Professor Wendy Duff, University of Toronto
Mr Mark Rogers, IP Australia
Mr Chris Hurley, Commonwealth Bank
Dr Richard Marciano, San Diego Super Computer Center
Mr Andrew Waugh, Public Record Office, Victoria
Mr David Kennedy
Dr Sergio Viademonte (September 2005-June 2006)
Dr Kate Cumming
Mr Malcolm Todd, National Archives UK
Ms Judith Ellis, Enterprise Knowledge Pty Ltd
Mr Hans Hofman, Nationaal Archief, The Hague
Mr Edward Hay, Deputy Auditor General, Victoria
Dr Andrew Wood, DSTC (November 2003-July 2005)
Dr Nigel Ward, Australian ADL Partnership Laboratory
National Archives of Australia
The National Archives provided expert staff contributing to the design of the proof of concept prototype in both iterations, identifying and documenting business case and risk frameworks, and providing vital links to the Government Online, metadata, software development, and records communities. In particular, National Archives staff contributed their knowledge of relevant government metadata standards and government business drivers to help maximise the practical relevance of the research work. Adrian Cunningham, from the NAA, convened and chaired the Advisory Committee, consisting of representatives from government agencies, the software industry, the records and archives profession, and metadata and standards setting communities.
State Records NSW
State Records NSW provided expert staff contributing guidance through participation in the Steering and Advisory Committees.
Australian Society of Archivists, Descriptive Standards Committee
The ASA Descriptive Standards Committee provided support to the project by the involvement of one of its members in the Steering and Advisory Committees. It provided expert advice to the Research Team. It will also facilitate the distribution and dissemination of research findings to national and international descriptive standards bodies.
Collaborations and Alliances
The proposed project is a continuation of long standing collaboration between the National Archives (NAA), the State Records Authority of New South Wales (SRA NSW), the Australian Society of Archivists (ASA), and Monash University, particularly in the area of identifying and standardising metadata requirements for recordkeeping, a research partnership which was strengthened and extended by involvement in the 1998-99 SPIRT Project which developed the Australian Recordkeeping Metadata Schema and recognised as of world-leading significance in efforts to devise durable recordkeeping strategies that meet the challenges of the digital age. This Linkage project further strengthened and extended research collaboration between the industry and higher education sectors. This particular project grew from a Research Forum involving NAA, SRA NSW, ASA Descriptive Standards Committee and other industry stakeholders. The objective of the meeting was to identify areas for metadata research and development which would meet high priority industry sector needs.
InterPARES 2 Description Cross Domain
A formal collaborative agreement with the Interpares2 Description Cross Domain group and the Clever Recordkeeping Metadata project resulted in collaboration on work on Metadata Repositories undertaken in the Interpares2 project. Ms Joanne Evans worked with research students at UCLA on the MADRAS initiative of the Interpares2 research project.
The Research Team workshopped and contributed to the development of IDEF models ‘Managing the Records Continuum’ of Mr Hans Hofman, work undertaken within the Interpares2 project.
IT 21/7 Recordkeeping Metadata Committee
Professor Sue McKemmish, Joanne Evans, Barbara Reed, Adrian Cunningham, Karen Skelton, Andrew Waugh and Tony Leviston were active contributors to the work of the Standards Australia Recordkeeping Metadata Committee under the auspices of IT 21, Recordkeeping. The successful outcomes of the work from this Committee were fast tracked to the international standards arena and has been incorporated into ISO/DTS 23081-2, 2007, Information and documentation -- Records management processes -- Metadata for records -- Part 2: Concepts and implementation issues.
ISO TC 46/SC11 WG 1: Records Management Metadata
This working group of the International Standard Organisation’s Sub Committee on Records Management is focused on developing standards for Records Metadata. The work of IT 21/7, which has been incorporated into ISO/DTS 23081-2, 2007, Information and documentation -- Records management processes -- Metadata for records -- Part 2: Concepts and implementation issues is detailed above. In addition Joanne Evans, in collaboration with colleagues from UCLA, has developed a recordkeeping metadata schema evaluation tool which is being considered for adoption as Part 3 of the ISO 23081 suite of standards.
Advisory Group Briefings
The CRKM Advisory Group met on four occasions throughout the project. At each meeting comprehensive overview briefings were presented to the Committee detailing progress and challenges encountered. These briefings offer a history of the development of the research project:-
- Briefing to Advisory Committee November 2004
- Briefing to Advisory Committee April 2005
- Briefing to Advisory Committee October 2005
- Briefing to Advisory Committee May 2006
Joanne Evans, Barbara Reed and Sue McKemmish, 'Interoperable Data: Sustainable Frameworks for Creating and Managing Recordkeeping Metadata', Records Management Journal, vol. 18, no. 2, 2008, pp. 115-129.
Barbara Reed, 'Service Oriented Architectures and Recordkeeping', Records Management Journal, vol. 18, no. 1, 2008, pp. 7-20.
Joanne Evans, 'Evaluating the Recordkeeping Capabilities of Metadata Schemas', Archives & Manuscripts, vol. 35, no. 2, November 2007, pp. 56-84.
Joanne Evans, Sue McKemmish and Karuna Bhoday, 'Create Once, Use Many Times: The Clever Use of Recordkeeping Metadata for Multiple Archival Purposes', Archival Science, vol. 5, no. 1, March 2005, pp. 17-42.
Anne Gilliland, Nadav Rouche, Lori Lindbergh and Joanne Evans, 'Towards a 21st Century Metadata Infrastructure Supporting the Creation, Preservation and Use of Trustworthy Records: Developing the InterPARES 2 Metadata Schema Registry', Archival Science, vol. 5, no. 1, March 2005, pp. 43-78.
Joanne Evans and Nadav Rouche, 'Utilizing Systems Development Methods in Archival Systems Research: Building a Metadata Schema Registry', Archival Science, vol. 4, no. 3-4, December 2004, pp. 315-334.
Joanne Evans and Sue McKemmish, Discovery through Innovation: Report on Outcomes from the Clever Recordkeeping Metadata Project, Presented at Archives: Discovery and Exploration, Annual Conference Australian Society of Archivists, Perth 6-9 August 2008.
Joanne Evans and Lori Lindberg, 'Describing and Analysing the Recordkeeping Capabilities of Metadata Sets', in DC-2004: Proceedings of the International Conference on Dublin Core and Metadata Applications October 11-14 2004 Shanghai, China, Shanghai Scientific and Technological Literature Publishing House, Shanghai, China, 2004, pp. 75-80. Also available at http://www.slais.ubc.ca/PEOPLE/faculty/tennis-p/dcpapers2004/Paper_27.pdf.
Joanne Evans, Sue McKemmish and Karuna Bhoday, 'Create Once, Use Many Times: The Clever Use of Recordkeeping Metadata for Multiple Archival Purposes', Presented at Smart Metadata and the Archives of the Future session at the International Council of Archives Congress, Vienna 23-28 August 2004. Paper and presentation available at http://www.wien2004.ica.org/fo/speakers.php?ctNv1=48&ctNv2=&IdSpk=240&AlphSpk=&p=&SpkV=2.
Presentations and Submissions
Joanne Evans, Sue McKemmish, and Barbara Reed, Making Metadata Matter: Outcomes from the Clever Recordkeeping Metadata Project, National Archives of Australia Professional Development Seminar, 21 October 2008, Canberra. (Pdf version of presentation with notes)
Joanne Evans and the CRKM Research Team, Making Metadata Matter: Outcomes from the Clever Recordkeeping Metadata Project, ASA Victorian Branch Seminar, 21 August 2008.
Joanne Evans, 'Create Once, Use Many Times': Clever Recordkeeping Metadata Research Project Report, Presented at National Archives of Australia Decade in Digital Session at ICA Congress 2008, Kuala Lumpur, 23 July 2008.
Joanne Evans, Metadata Registries for Recordkeeping Metadata Interoperability, 11th Open Forum on Metadata Registries, Sydney, Australia, 19-22 May 2008.
Barbara Reed, Service Oriented Architectures and Recordkeeping, Influence with Integrity, 24th International RMAA Convention, 9 to 13 September 2007, Wellington - New Zealand.
CRKM Project Team, Submission to Australian Digital Records Initiative, Digital Records Export Standard, 21 August 2006.
Joanne Evans, Sue McKemmish, Karuna Bhoday, Andi Sumartono and Sergio Viademonte, Investigating Metadata Interoperability in the Clever Recordkeeping Metadata Project, presentation to IMS 5010 Evidence and Metadata Students, Monash University Caulfield, 30 May 2006.
Sue McKemmish, Karuna Bhoday, Joanne Evans, Andi Sumartono and Sergio Viademonte, Investigating Metadata Interoperability in the Clever Recordkeeping Metadata Project, presentation to Australian Government Information Management Office (AGIMO), Interoperability Group, Canberra, 6 March 2006.
Karuna Bhoday, Joanne Evans, Sue McKemmish, Andi Sumartono and Sergio Viademonte, Create Once, Use Many Times: The Clever Use of Metadata in eGovernment and eBusiness Recordkeeping Processes in Networked Environments, Archives and Communities, ASA/ARANZ 2005 Conference New Zealand, October 2005.
Karuna Bhoday, Joanne Evans, Sue McKemmish, Andi Sumartono and Sergio Viademonte, Create Once, Use Many Times: Investigating Interoperability in the CRKM Project, Recordkeeping Metadata Seminar, ASA/ARANZ Archives and Communities 2005 Conference New Zealand, 6 October 2005.
Joanne Evans, Karuna Bhoday, Sue McKemmish and Sergio Viademonte, Automating Recordkeeping Metadata Capture and Re-use: Translations from Theory to Practice, Association of Canadian Archivists Conference 2005, June 8-11, Saskatoon, Canada.
Karuna Bhoday and Joanne Evans, Presentation to Recordkeeping Metadata Information and Briefing Seminar, Melbourne 5 November 2004.
Joanne Evans, Presentation of Metadata Schema Registry to ISO 23081 Working Group, Sydney November 2004.
Australian Government Locator Service (AGLS) - Metadata in the form of metatags to provide better access to government information online and facilitate eGovernment.
Descriptive metadata - Structured or semi-structured information that enables the creation, management and use of records through time, and within and across the domains in which they are created and used.
eBusiness - involves the use of IT and internet-based technologies (such as intranets and extranets) to support commerce and improve business performance.
eGovernment - the application of new technologies to government services, information and administration, to demonstrate sustained benefits to citizens, business and government itself.
Source: NOIE, Better Services, Better Government: the Federal Government's E-government Strategy, 2002.
Implementation modelling - Implementation modelling involves simulating the implementation of a system in order to develop a framework that can be used to guide further implementations.
InterPARES - International Research on Permanent Authentic Records in Electronic Systems project, funded by the Social Sciences and Humanities Research Council, of Canada.
Metadata - generically defined as "structured data about data".
Metadata Broker - A service that enables the translation of metadata between schemas.
Metadata registry - A metadata registry is a formal system that records the semantics, structure, and interchange formats of any type of data, e.g. data found in databases, messages, documents and other applications. The descriptions of data are recorded in the form of metadata (data about data). ... One of the essential characteristics of a metadata registry is the existence of a formal Authority agency that manages development and evolution of the registry and is responsible for policies pertaining to contents and operation of the registry. An attribute of each set of metadata should be the authority agency for that particular set.
Source: Joint Workshop on Metadata Registries, Draft Workshop Report 1.6, 23 December 1997.
Metadata standard - A codification of metadata requirements from a particular perspective. A metadata standard defines both the object(s) and the data about those object(s) to satisfy the requirements of a particular community or viewpoint.
Metatools - Metatools are tools to support the capture, management and use of metadata. They may include metadata registries, repositories and translations and transformation services.
Proof of concept prototype - In the context of the Clever Recordkeeping Metadata Project, the proof of concept prototype will be the model of a system which illustrates the automated capture and re-use of recordkeeping metadata.
Recordkeeping - Making and maintaining complete, accurate and reliable evidence of business transactions in the form of recorded information.
Source: AS 4390-1996 Australian Standard: Records Management, Standards Australia, Homebush, 1996.
Recordkeeping Metadata - Structured or semi-structured information which enables the creation, management and use of records through time and within and across domains in which they are created. Recordkeeping metadata can be used to identify, authenticate, and contextualize records; and the people, processes and systems that create, manage, maintain and use them.
Source: David Wallace, 'Archiving Metadata Forum: Report from the Recordkeeping Metadata Working Meeting, June 2000', Archival Science, vol. 1, no. 3, 2001, pp. 253-269.
Service Oriented Architecture - A SOA (Service Oriented Architecture) is a component model that inter-relates the different functional units of an application, called services, through well-defined interfaces and contracts between these services. The interface is defined in a neutral manner that should be independent of the hardware platform, the operating system, and the programming language the service is implemented in. This allows services, built on a variety of such systems, to interact with each other in a uniform and universal manner.
Source: IBM developerWorks, New to SOA and Web Services, 5 November 2004.
User-centred rapid prototyping - User-centred rapid prototyping involves the development of a prototype in a series of short iterations with the user playing a central role in the design and evaluation of the system.
Web services - A Web service is a software system identified by a URI [RFC 2396], whose public interfaces and bindings are defined and described using XML. Its definition can be discovered by other software systems. These systems may then interact with the Web service in a manner prescribed by its definition, using XML based messages conveyed by Internet protocols.
Source: W3C, Web Services Glossary, W3C Working Draft 14 November 2002, Allen Brown and Hugo Haas (editors), W3C, 2002, http://www.w3.org/TR/2002/WD-ws-gloss-20021114/.