Archives and AI: An Overview of Current Debates and Future Perspectives

The digital transformation is turning archives, both old and new, into data. As a consequence, automation in the form of artificial intelligence techniques is increasingly applied both to scale traditional recordkeeping activities, and to experiment with novel ways to capture, organise, and access records. We survey recent developments at the intersection of Artificial Intelligence and archival thinking and practice. Our overview of this growing body of literature is organised through the lenses of the Records Continuum model. We find four broad themes in the literature on archives and artificial intelligence: theoretical and professional considerations, the automation of recordkeeping processes, organising and accessing archives, and novel forms of digital archives. We conclude by underlining emerging trends and directions for future work, which include the application of recordkeeping principles to the very data and processes that power modern artificial intelligence and a more structural - yet critically aware - integration of artificial intelligence into archival systems and practice

Hierarchical distributed fog-to-cloud data management in smart cities

There is a vast amount of data being generated every day in the world with different formats, quality levels, etc. This new data, together with the archived historical data, constitute the seed for future knowledge discovery and value generation in several fields of science and big data environments. Discovering value from data is a complex computing process where data is the key resource, not only during its processing, but also during its entire life cycle. However, there is still a huge concern about how to organize and manage this data in all fields for efficient usage and exploitation during all data life cycles. Although several specific Data LifeCycle (DLC) models have been recently defined for particular scenarios, we argue that there is no global and comprehensive DLC framework to be widely used in different fields. In particular scenario, smart cities are the current technological solutions to handle the challenges and complexity of the growing urban density. Traditionally, Smart City resources management rely on cloud based solutions where sensors data are collected to provide a centralized and rich set of open data. The advantages of cloud-based frameworks are their ubiquity, as well as an (almost) unlimited resources capacity. However, accessing data from the cloud implies large network traffic, high latencies usually not appropriate for real-time or critical solutions, as well as higher security risks. Alternatively, fog computing emerges as a promising technology to absorb these inconveniences. It proposes the use of devices at the edge to provide closer computing facilities and, therefore, reducing network traffic, reducing latencies drastically while improving security. We have defined a new framework for data management in the context of a Smart City through a global fog to cloud resources management architecture. This model has the advantages of both, fog and cloud technologies, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. In this thesis, we propose many novel ideas in the design of a novel F2C Data Management architecture for smart cities as following. First, we draw and describe a comprehensive scenario agnostic Data LifeCycle model successfully addressing all challenges included in the 6Vs not tailored to any specific environment, but easy to be adapted to fit the requirements of any particular field. Then, we introduce the Smart City Comprehensive Data LifeCycle model, a data management architecture generated from a comprehensive scenario agnostic model, tailored for the particular scenario of Smart Cities. We define the management of each data life phase, and explain its implementation on a Smart City with Fog-to-Cloud (F2C) resources management. And then, we illustrate a novel architecture for data management in the context of a Smart City through a global fog to cloud resources management architecture. We show this model has the advantages of both, fog and cloud, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. As a first experiment for the F2C data management architecture, a real Smart City is analyzed, corresponding to the city of Barcelona, with special emphasis on the layers responsible for collecting the data generated by the deployed sensors. The amount of daily sensors data transmitted through the network has been estimated and a rough projection has been made assuming an exhaustive deployment that fully covers all city. And, we provide some solutions to both reduce the data transmission and improve the data management. Then, we used some data filtering techniques (including data aggregation and data compression) to estimate the network traffic in this model during data collection and compare it with a traditional real system. Indeed, we estimate the total data storage sizes through F2C scenario for Barcelona smart citiesAl món es generen diàriament una gran quantitat de dades, amb diferents formats, nivells de qualitat, etc. Aquestes noves dades, juntament amb les dades històriques arxivades, constitueixen la llavor per al descobriment de coneixement i la generació de valor en diversos camps de la ciència i grans entorns de dades (big data). Descobrir el valor de les dades és un procés complex de càlcul on les dades són el recurs clau, no només durant el seu processament, sinó també durant tot el seu cicle de vida. Tanmateix, encara hi ha una gran preocupació per com organitzar i gestionar aquestes dades en tots els camps per a un ús i explotació eficients durant tots els cicles de vida de les dades. Encara que recentment s'han definit diversos models específics de Data LifeCycle (DLC) per a escenaris particulars, argumentem que no hi ha un marc global i complet de DLC que s'utilitzi àmpliament en diferents camps. En particular, les ciutats intel·ligents són les solucions tecnològiques actuals per fer front als reptes i la complexitat de la creixent densitat urbana. Tradicionalment, la gestió de recursos de Smart City es basa en solucions basades en núvol (cloud computing) on es recopilen dades de sensors per proporcionar un conjunt de dades obert i centralitzat. Les avantatges dels entorns basats en núvol són la seva ubiqüitat, així com una capacitat (gairebé) il·limitada de recursos. Tanmateix, l'accés a dades del núvol implica un gran trànsit de xarxa i, en general, les latències elevades no són apropiades per a solucions crítiques o en temps real, així com també per a riscos de seguretat més elevats. Alternativament, el processament de boira (fog computing) sorgeix com una tecnologia prometedora per absorbir aquests inconvenients. Proposa l'ús de dispositius a la vora per proporcionar recuirsos informàtics més propers i, per tant, reduir el trànsit de la xarxa, reduint les latències dràsticament mentre es millora la seguretat. Hem definit un nou marc per a la gestió de dades en el context d'una ciutat intel·ligent a través d'una arquitectura de gestió de recursos des de la boira fins al núvol (Fog-to-Cloud computing, o F2C). Aquest model té els avantatges combinats de les tecnologies de boira i de núvol, ja que permet reduir les latències per a aplicacions crítiques mentre es poden utilitzar les grans capacitats informàtiques de la tecnologia en núvol. En aquesta tesi, proposem algunes idees noves en el disseny d'una arquitectura F2C de gestió de dades per a ciutats intel·ligents. En primer lloc, dibuixem i descrivim un model de Data LifeCycle global agnòstic que aborda amb èxit tots els reptes inclosos en els 6V i no adaptats a un entorn específic, però fàcil d'adaptar-se als requisits de qualsevol camp en concret. A continuació, presentem el model de Data LifeCycle complet per a una ciutat intel·ligent, una arquitectura de gestió de dades generada a partir d'un model agnòstic d'escenari global, adaptat a l'escenari particular de ciutat intel·ligent. Definim la gestió de cada fase de la vida de les dades i expliquem la seva implementació en una ciutat intel·ligent amb gestió de recursos F2C. I, a continuació, il·lustrem la nova arquitectura per a la gestió de dades en el context d'una Smart City a través d'una arquitectura de gestió de recursos F2C. Mostrem que aquest model té els avantatges d'ambdues, la tecnologia de boira i de núvol, ja que permet reduir les latències per a aplicacions crítiques mentre es pot utilitzar la gran capacitat de processament de la tecnologia en núvol. Com a primer experiment per a l'arquitectura de gestió de dades F2C, s'analitza una ciutat intel·ligent real, corresponent a la ciutat de Barcelona, amb especial èmfasi en les capes responsables de recollir les dades generades pels sensors desplegats. S'ha estimat la quantitat de dades de sensors diàries que es transmet a través de la xarxa i s'ha realitzat una projecció aproximada assumint un desplegament exhaustiu que cobreix tota la ciutat

Hierarchical distributed fog-to-cloud data management in smart cities

There is a vast amount of data being generated every day in the world with different formats, quality levels, etc. This new data, together with the archived historical data, constitute the seed for future knowledge discovery and value generation in several fields of science and big data environments. Discovering value from data is a complex computing process where data is the key resource, not only during its processing, but also during its entire life cycle. However, there is still a huge concern about how to organize and manage this data in all fields for efficient usage and exploitation during all data life cycles. Although several specific Data LifeCycle (DLC) models have been recently defined for particular scenarios, we argue that there is no global and comprehensive DLC framework to be widely used in different fields. In particular scenario, smart cities are the current technological solutions to handle the challenges and complexity of the growing urban density. Traditionally, Smart City resources management rely on cloud based solutions where sensors data are collected to provide a centralized and rich set of open data. The advantages of cloud-based frameworks are their ubiquity, as well as an (almost) unlimited resources capacity. However, accessing data from the cloud implies large network traffic, high latencies usually not appropriate for real-time or critical solutions, as well as higher security risks. Alternatively, fog computing emerges as a promising technology to absorb these inconveniences. It proposes the use of devices at the edge to provide closer computing facilities and, therefore, reducing network traffic, reducing latencies drastically while improving security. We have defined a new framework for data management in the context of a Smart City through a global fog to cloud resources management architecture. This model has the advantages of both, fog and cloud technologies, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. In this thesis, we propose many novel ideas in the design of a novel F2C Data Management architecture for smart cities as following. First, we draw and describe a comprehensive scenario agnostic Data LifeCycle model successfully addressing all challenges included in the 6Vs not tailored to any specific environment, but easy to be adapted to fit the requirements of any particular field. Then, we introduce the Smart City Comprehensive Data LifeCycle model, a data management architecture generated from a comprehensive scenario agnostic model, tailored for the particular scenario of Smart Cities. We define the management of each data life phase, and explain its implementation on a Smart City with Fog-to-Cloud (F2C) resources management. And then, we illustrate a novel architecture for data management in the context of a Smart City through a global fog to cloud resources management architecture. We show this model has the advantages of both, fog and cloud, as it allows reduced latencies for critical applications while being able to use the high computing capabilities of cloud technology. As a first experiment for the F2C data management architecture, a real Smart City is analyzed, corresponding to the city of Barcelona, with special emphasis on the layers responsible for collecting the data generated by the deployed sensors. The amount of daily sensors data transmitted through the network has been estimated and a rough projection has been made assuming an exhaustive deployment that fully covers all city. And, we provide some solutions to both reduce the data transmission and improve the data management. Then, we used some data filtering techniques (including data aggregation and data compression) to estimate the network traffic in this model during data collection and compare it with a traditional real system. Indeed, we estimate the total data storage sizes through F2C scenario for Barcelona smart citiesAl món es generen diàriament una gran quantitat de dades, amb diferents formats, nivells de qualitat, etc. Aquestes noves dades, juntament amb les dades històriques arxivades, constitueixen la llavor per al descobriment de coneixement i la generació de valor en diversos camps de la ciència i grans entorns de dades (big data). Descobrir el valor de les dades és un procés complex de càlcul on les dades són el recurs clau, no només durant el seu processament, sinó també durant tot el seu cicle de vida. Tanmateix, encara hi ha una gran preocupació per com organitzar i gestionar aquestes dades en tots els camps per a un ús i explotació eficients durant tots els cicles de vida de les dades. Encara que recentment s'han definit diversos models específics de Data LifeCycle (DLC) per a escenaris particulars, argumentem que no hi ha un marc global i complet de DLC que s'utilitzi àmpliament en diferents camps. En particular, les ciutats intel·ligents són les solucions tecnològiques actuals per fer front als reptes i la complexitat de la creixent densitat urbana. Tradicionalment, la gestió de recursos de Smart City es basa en solucions basades en núvol (cloud computing) on es recopilen dades de sensors per proporcionar un conjunt de dades obert i centralitzat. Les avantatges dels entorns basats en núvol són la seva ubiqüitat, així com una capacitat (gairebé) il·limitada de recursos. Tanmateix, l'accés a dades del núvol implica un gran trànsit de xarxa i, en general, les latències elevades no són apropiades per a solucions crítiques o en temps real, així com també per a riscos de seguretat més elevats. Alternativament, el processament de boira (fog computing) sorgeix com una tecnologia prometedora per absorbir aquests inconvenients. Proposa l'ús de dispositius a la vora per proporcionar recuirsos informàtics més propers i, per tant, reduir el trànsit de la xarxa, reduint les latències dràsticament mentre es millora la seguretat. Hem definit un nou marc per a la gestió de dades en el context d'una ciutat intel·ligent a través d'una arquitectura de gestió de recursos des de la boira fins al núvol (Fog-to-Cloud computing, o F2C). Aquest model té els avantatges combinats de les tecnologies de boira i de núvol, ja que permet reduir les latències per a aplicacions crítiques mentre es poden utilitzar les grans capacitats informàtiques de la tecnologia en núvol. En aquesta tesi, proposem algunes idees noves en el disseny d'una arquitectura F2C de gestió de dades per a ciutats intel·ligents. En primer lloc, dibuixem i descrivim un model de Data LifeCycle global agnòstic que aborda amb èxit tots els reptes inclosos en els 6V i no adaptats a un entorn específic, però fàcil d'adaptar-se als requisits de qualsevol camp en concret. A continuació, presentem el model de Data LifeCycle complet per a una ciutat intel·ligent, una arquitectura de gestió de dades generada a partir d'un model agnòstic d'escenari global, adaptat a l'escenari particular de ciutat intel·ligent. Definim la gestió de cada fase de la vida de les dades i expliquem la seva implementació en una ciutat intel·ligent amb gestió de recursos F2C. I, a continuació, il·lustrem la nova arquitectura per a la gestió de dades en el context d'una Smart City a través d'una arquitectura de gestió de recursos F2C. Mostrem que aquest model té els avantatges d'ambdues, la tecnologia de boira i de núvol, ja que permet reduir les latències per a aplicacions crítiques mentre es pot utilitzar la gran capacitat de processament de la tecnologia en núvol. Com a primer experiment per a l'arquitectura de gestió de dades F2C, s'analitza una ciutat intel·ligent real, corresponent a la ciutat de Barcelona, amb especial èmfasi en les capes responsables de recollir les dades generades pels sensors desplegats. S'ha estimat la quantitat de dades de sensors diàries que es transmet a través de la xarxa i s'ha realitzat una projecció aproximada assumint un desplegament exhaustiu que cobreix tota la ciutat.Postprint (published version

Observations and Recommendations: on the Republic of Ireland Open Data Strategy (Working Paper 3)

1. Introduction - Context The Republic of Ireland Department of Public Expenditures and Reform (DPER) launched its first Open Data Portal data.gov.ie on July 22nd. It also conducted three public consultations, one with public sector bodies on September 22nd, and later on September 8th with civil society organizations and the public, while also accepting responses and recommendations by email. Open Data in the Republic is being led and coordinated by the DPER Government Reform Unit and the Chief Information Officer. The new CKAN portal was created by Insight Galway and it marks an important milestone toward meeting the following government objectives to: • enhance openness and transparency, • strengthen public governance and • yield economic benefits. The following are the information products delivered by Insight at National University of Ireland at Galway (NUIG): 1. Best Practice Handbook 2. Data Audit Report 3. Roadmap 4. Evaluation Framework 5. Open Data Publication Handbook Information products were delivered as per the DPER Request for Tender (RfT) and according to the following high level requirements: • Provide advice and support to facilitate the implementation of key objectives and actions related to the Open Data initiative. • Identify and align with national requirements best practice and international standards for Open Data; • Help specify short- , medium- and long-term objectives for Open Data in Ireland; • Identify potential opportunities for Ireland to design its Open Data drawing on the experience of what has worked and has not worked in other jurisdictions and also drawing on the scope for positively differentiating Open Data in Ireland; • Establish an inventory of datasets currently available in the public domain as well as assist in identifying priority areas for the publication of datasets in the future; and • Develop a roadmap for how Ireland could achieve its objectives in the area of Open Data. In the spirit of collaboration and openness, after the July 22nd launch the DPER solicited input on the following questions from public sector bodies that have expertise and a stake in the initiative: • What do you see as the priorities for Open Data and next steps? • Any issues/concerns that need to be addressed in standardisation and publication of data? • How we would maintain this type of group going forward given the level of expertise and experience in the room today – maybe meet as a Forum 3 times a year? • Indicate whether you are willing to participate in a smaller group and work in an engaged way to support DPER on particular aspects and on the development of an Open Data strategy in the coming months? • How we should engage with civil society, business interests and other users to ensure we focus our attention on high value datasets that will be used so that potential economic, social and political benefits can be realised? Finally, the DPER solicited input from the public but unfortunately submissions were to be made to an email address in lieu of an open forum whereby others could see who and what was submitted. This launch marks a turning point in Ireland’s open data story which was sketched out in the 1st of this 3 part series on Open Data and Open Government. This milestone affirms Ireland’s international commitment to the Open Government Partnership (OGP) and it is also a precondition for Ireland’s ability to sign onto the G8 Open Data Charter whereby signatories commit to: • Open Data by Default • Quality and Quantity • Useable by All • Releasing Data for Improved Governance • Releasing Data for Innovation OGP membership, signing onto the G8 Charter and the launch of the Open Data portal, are positive steps toward the advancement of public sector reform and Minister Howlin, DPER and Insight are to be lauded for moving this plan forward. The rest of this paper examines the DPER Open Data plan, the DPER/Insight portal, research documents, their recommendations and provides answers to a series of questions DPER officials posed after the July 22, 2014 public sector consultations. It is also The Programmable City Project submission to the DPER Open Data Consultation

Observations and Recommendations: on the Republic of Ireland Open Data Strategy (Working Paper 3)

1. Introduction - Context The Republic of Ireland Department of Public Expenditures and Reform (DPER) launched its first Open Data Portal data.gov.ie on July 22nd. It also conducted three public consultations, one with public sector bodies on September 22nd, and later on September 8th with civil society organizations and the public, while also accepting responses and recommendations by email. Open Data in the Republic is being led and coordinated by the DPER Government Reform Unit and the Chief Information Officer. The new CKAN portal was created by Insight Galway and it marks an important milestone toward meeting the following government objectives to: • enhance openness and transparency, • strengthen public governance and • yield economic benefits. The following are the information products delivered by Insight at National University of Ireland at Galway (NUIG): 1. Best Practice Handbook 2. Data Audit Report 3. Roadmap 4. Evaluation Framework 5. Open Data Publication Handbook Information products were delivered as per the DPER Request for Tender (RfT) and according to the following high level requirements: • Provide advice and support to facilitate the implementation of key objectives and actions related to the Open Data initiative. • Identify and align with national requirements best practice and international standards for Open Data; • Help specify short- , medium- and long-term objectives for Open Data in Ireland; • Identify potential opportunities for Ireland to design its Open Data drawing on the experience of what has worked and has not worked in other jurisdictions and also drawing on the scope for positively differentiating Open Data in Ireland; • Establish an inventory of datasets currently available in the public domain as well as assist in identifying priority areas for the publication of datasets in the future; and • Develop a roadmap for how Ireland could achieve its objectives in the area of Open Data. In the spirit of collaboration and openness, after the July 22nd launch the DPER solicited input on the following questions from public sector bodies that have expertise and a stake in the initiative: • What do you see as the priorities for Open Data and next steps? • Any issues/concerns that need to be addressed in standardisation and publication of data? • How we would maintain this type of group going forward given the level of expertise and experience in the room today – maybe meet as a Forum 3 times a year? • Indicate whether you are willing to participate in a smaller group and work in an engaged way to support DPER on particular aspects and on the development of an Open Data strategy in the coming months? • How we should engage with civil society, business interests and other users to ensure we focus our attention on high value datasets that will be used so that potential economic, social and political benefits can be realised? Finally, the DPER solicited input from the public but unfortunately submissions were to be made to an email address in lieu of an open forum whereby others could see who and what was submitted. This launch marks a turning point in Ireland’s open data story which was sketched out in the 1st of this 3 part series on Open Data and Open Government. This milestone affirms Ireland’s international commitment to the Open Government Partnership (OGP) and it is also a precondition for Ireland’s ability to sign onto the G8 Open Data Charter whereby signatories commit to: • Open Data by Default • Quality and Quantity • Useable by All • Releasing Data for Improved Governance • Releasing Data for Innovation OGP membership, signing onto the G8 Charter and the launch of the Open Data portal, are positive steps toward the advancement of public sector reform and Minister Howlin, DPER and Insight are to be lauded for moving this plan forward. The rest of this paper examines the DPER Open Data plan, the DPER/Insight portal, research documents, their recommendations and provides answers to a series of questions DPER officials posed after the July 22, 2014 public sector consultations. It is also The Programmable City Project submission to the DPER Open Data Consultation

Small Pool for Big Data: Researching for Sustainable Data Focused on Open Government Data (OGD) Movement

When Sir Isaac Newton said his famous statement standing on the shoulders of giants, it was a modest phrase and explained the necessity of sharing knowledge or information to make the next intellectual progress. The data industry is now the fastest developing area, but many ambiguities are a subject in law. The protection of data is a fascinating and still unsolved challenge for intellectual property law. Data is essential in the matter of new industry and our lifestyle at individual, corporate, and institutional levels. And the legal protection needs to work to offer vivid transactions of data for creative interactions. However, many enterprises consider data an asset for business profit as the data industry grows vast and fast. Data raises diverse policy debates that arise in the better-known intellectual property areas, for instance, copyrights, unfair competition, and trade secret. The vague aspects of data implicate a number of intellectual property approaches. It also extends to the economic problem \u27tragedy of anti-commons\u27 that fragmented ownership is disrupting sound usage. In this regard, Open Governmental Data (OGD) is one way to resolve inefficiency in the data industry. The government collects massive personal data and reproduces datasets in the process of administration. Many governments give back the public data for private sectors anticipating the data works for new enterprise seed money. This work looks at three considerations about the legal aspects of data. At first, we will see the necessity of big data in current and reasons for the government to pay attention to open data to the public. The data industry market\u27s inefficiency discourages cumulative innovation in our society and approaches the benefits of sharing data in the private economy or OGD movement. Second, the paper conducts principles of OGD and takes a functional approach in analyzing the related IP laws in database protection and public accessibility. Interestingly various governments are opening data that compares various OGD models from different countries led by other stakeholders, including government, large companies, small to medium enterprises ( SMEs ), and how they work as a member of OGD. Finally, it critiques the current OGD movement and suggests that corporate OGD strategies granting autonomous would help resolve the anti-commons of IP in the big data industry

Towards a circular building industry through digitalisation

This thesis explores the integration of Circular Economy (CE) principles of narrow, slow, close, and regenerate in the social housing practice through digital technologies. Beginning with the examination of the CE implementation in Dutch social housing organisations, the research extends its focus to the broader built environment, introducing the Circular Digital Built Environment Framework and identifying ten enabling technologies. Subsequent chapters explore realworld applications of these digital technologies in circular new built, renovation, maintenance, and demolition projects of forerunner social housing organisations. The thesis includes a comprehensive study of material passports, addressing challenges around data management and proposing a digitally-enabled framework. The thesis concludes with critical reflections on the findings and their implications and provides further recommendations for research and practical applications in advancing circularity in the building industry through digital technologies