Tag Archives: web archiving

Internet Archive Launches Collaborative, Web-Based Art Resources Preservation and Access Initiative

Much of the art gallery, artist, and arts organization materials that were once published in print form are now available primarily or solely on the web. These groups, like many in the cultural sector, have also been hit especially hard by the global pandemic, making their web presences particularly at-risk of being lost if they are not proactively collected and preserved.The creation of reference and research resources that promote streamlined access and enable new types of scholarly use will ensure that the art historical record of the 21st century, and especially of our current global pandemic, is readily accessible far into the future.

For this reason, the Internet Archive, along with the New York Art Resources Consortium (NYARC), are pleased to announce our project Consortial Action to Preserve Born-Digital, Web-Based Art History & Culture. The project recently received a two-year, $305,343 Humanities Collections and Reference Resources grant from the Division of Preservation and Access at the National Endowment for the Humanities. This award will support the formation of a cooperative group of 30+ art and museum libraries from across the United States to collaborate on the preservation of, and access to vital arts content from the web. 

The Internet Archive has a long history of building and supporting collaborative communities and providing non-profit web, preservation, and access services to cultural heritage organizations. The multi-institutional initiative between Internet Archive, NYARC, and other arts and museum organizations will build on similar community-based archiving and professional cultivation projects in the Community Programs group, especially our Community Webs program, currently expanding nationally and internationally. Community Webs has received funding from The Andrew W. Mellon Foundation and IMLS to provide public libraries and cultural heritage organizations with services, training, and professional development opportunities to document their diverse local history. 

NYARC are pioneers in collaborative web archiving and shared services, among art and museum libraries. NYARC’s robust web archive collections encompass art resources, artists’ websites, auction catalogs, catalogues raisonnes, and hundreds of New York City gallery websites. The Internet Archive and NYARC have partnered on work to build born-digital collecting capacity among arts organizations in the past, most recently in the IMLS-funded Advancing Art Libraries and Curated Web Archives National forum and related events.  Through discussions, workshops and roadmapping sessions with leaders in art and museum libraries, a strategy and plan  towards an inclusive, sustainable, cooperative approach to collecting and stewarding born-digital, locally-focused art history collection was developed, forming the basis of this broader cooperative effort.

Members in the project’s preliminary group of art and museum libraries will select topics and specific web content that is relevant to their expertise, will provide metadata to facilitate access to archived content, and will participate in planning and evaluation meetings, all while curating a valuable reference resource that will enhance their traditional collecting areas. The Internet Archive will coordinate communications, facilitate governance and collective curatorial activities, provide technical digital library and archive services, and help enable members to build and maintain discovery and access platforms, as well as facilitate researcher use of the collections resulting from the group’s work.

If your art or museum library is interested in joining this collaborative effort, please fill out this participation form by July 31 to join us! 

Community Webs joins the Digital Public Library of America

Internet Archive’s Community Webs program is delighted to announce a partnership with the Digital Public Library of America (DPLA) to ingest metadata from the over 700 publicly available Community Webs web archive collections into DPLA. These collections include thousands of archived websites and millions of individual web-published resources that document local history and underrepresented groups. The Internet Archive has been a DPLA content provider since 2015, primarily contributing content from our many print digitizing partnerships. Community Webs will also join DPLA as a member and we are excited for this opportunity to add hyperlocal born-digital and web collections from public libraries nationwide into DPLA’s national portal to cultural heritage collections.

The Community Webs program was launched in 2017 to provide training, infrastructure, services, and professional community cultivation for public librarians across the country for the purpose of documenting local history and community archiving, especially documenting communities and populaces traditionally excluded from the historical record. The program is in the midst of nationwide expansion and currently includes more than 100 member public libraries who are collaborating with local organizations, movements, and groups to document the lives and accomplishments of their citizens. The program continues to add new public libraries and cultural heritage organizations to support and scale their community archiving and has an open call for applications in the US, Canada, and internationally for additional public libraries and local heritage organizations to join the program. Examples of Community Webs collections include:

  • Community Webs members have created more than 30 collections documenting local responses to the COVID-19 pandemic, including COVID-19 Coronavirus East Baton Rouge Parish from East Baton Rouge Parish Library and Schomburg Center for Research in Black Culture’s “Novel Coronavirus COVID-19” collection which focuses on “the African diasporan experiences of COVID-19 including racial disparities in health outcomes and access, the impact on Black-owned businesses, and cultural production.” 
  • Community Webs members have created a number of collections documenting LGBTQ groups, events and other resources, including LGBTQIA/Hormel Resources from San Francisco Public Library and Birmingham Public Library’s “LGBTQ in Alabama” collection.
  • Members are also actively archiving materials on their local or regional culture, such as Kansas City Public Library’s Arts & Culture collection, which “documents Kansas City’s thriving arts community, including galleries, museums, nonprofits, advocacy organizations, criticism and art spaces.”
  • Many members have focused on documenting local social services or advocacy groups, such as Madison Public Library’s Racial Equity and Social Justice, Madison, WI collection of “organizations and non-profits that engage in public discourse on issues of racial equity and social justice.”

Working with a mission-aligned organization like DPLA and our shared values of collaboration, open access, and community empowerment made it an obvious fit for Community Webs member collections to also be available in DPLA. Some public libraries who are a part of the Community Webs program are also members of local or statewide DPLA content hubs, and already have digitized content available in DPLA.The partnership between DPLA and Community Webs will ensure that archived web and born-digital collections are accessible alongside similar digitized materials for seamless discovery and access for uses. Pairing Community Webs’ free archiving, infrastructure, education, and other services with DPLA’s aggregation tools, hubs networks, and its advocacy role will help expand national access and capacity for making primary sources, and a more diverse archival record, accessible to any online user,

“DPLA’s new partnership with the Community Webs program will help further our mission to provide free digital access to cultural heritage artifacts that inform a truly representative history of our nation, “ said Shaneé Yvette Murrain, director of community engagement for DPLA. “We are thrilled to be deepening our work with Internet Archive through a program so perfectly aligned with our organizations’ shared values.”

“Pairing the community web archives of 100+ public libraries and the cohort cultivation that are part of Community Webs with the national scope and professional networks native to DPLA is a perfect match. We are excited to expand access to these amazing grassroots digital collections,” said Jefferson Bailey, Director of Web Archiving & Data Services at Internet Archive.

We are excited to be partnering with DPLA to increase access to these vital community history collections and look forward to building more integrations and furthering this collaboration in the years to come.

Introducing 50+ New Public Library Members of the Internet Archive’s Community Webs Program

The Internet Archive’s Community Webs Program provides training and education, infrastructure and services, and professional community cultivation for public librarians across the country to document their local history and the lives of their patrons. Following our recent announcement of the program’s national expansion, with support from the Andrew W. Mellon Foundation, we are excited to welcome the first class of 50+ new public libraries to the program. This brings the current number of new and returning Community Webs participants to 90+ libraries from 33 states and 3 US territories. This diverse group of organizations includes multiple state libraries representing their regions, as well as a mix of large metropolitan library systems, small libraries in rural areas, and libraries like the Feleti Barstow Public Library in American Samoa. All will be working to document their communities, with a particular focus on archiving materials from traditionally underrepresented groups.

The new cohort class kicked off with virtual introductory events in mid-March, where participants met one another and shared stories about their communities and their goals for preserving and providing access to local history materials. Member libraries are currently receiving training in topics such as collection development and starting to build digital collections that reflect local diversity, events, and culture.

Program participant Kathleen Pickering, Director of the Belen Public Library and Harvey House Museum in Belen, New Mexico notes that their library “is committed to free and open-source electronic resources for our patrons, especially given the low-income status of many of our residents” and Community Webs will help further that goal. Similarly, new cohort member Aaron Ramirez of Pueblo City-County Library District (PCCLD) found Community Webs to be a great fit for existing institutional goals and initiatives. “PCCLD’s five-year strategic plan directs us to embrace local cultures, to include individuals of all skill levels and physical abilities, and to enrich established partnerships and collaborations. The groups that have not seen themselves in our archives will find through this project PCCLD’s intention and means to listen and go forward as allies and as a resource of support, rather than an institution serving only the affluent.”

Makiba J. Foster

Makiba J. Foster, Manager of The African American Research Library and Cultural Center of Broward County, Florida pointed out that “as content becomes increasingly digital, we need this opportunity to document the digital life and content of our community which includes a diverse representation of the Black Diaspora.”  Makiba was a member of the original Community Webs cohort in a previous position at the Schomburg Center for Research in Black Culture at New York Public Library, and recently presented on her work archiving the black diaspora to a group of more than 200 attendees.

The Community Webs Program is continuing to grow towards the milestone of over 150 participating libraries across the United States and will soon announce another call for applicants for a U.S. cohort starting in late summer. The program also is beginning to expand internationally, starting in Canada, exploring the addition of other types of libraries and cultural heritage organizations, and expanding its suite of training and services available to participants. Expect more news on these initiatives soon. 

Welcome to our new cohort of Community Webs libraries! The full list of new members: 

  • Alamogordo Public Library (New Mexico)
  • Amelia Island Museum of History (Florida)
  • ART | library deco (Texas)
  • Asbury Park Public Library (New Jersey)
  • Atlanta History Center (Georgia)
  • Bartholomew County Public Library (Indiana)
  • Bedford Public Library System (Virginia)
  • Belen Public Library and Harvey House Museum (New Mexico)
  • Bensenville Community Public Library (Illinois)
  • Biblioteca Municipal Aurea M. Pérez (Puerto Rico)
  • Carbondale Public Library (Illinois)
  • Cedar Mill & Bethany Community Libraries (Oregon)
  • Charlotte Mecklenburg Library (North Carolina)
  • Chicago Public Library (Illinois)
  • City Archives & Special Collections, New Orleans Public Library (Louisiana)
  • Dayton Metro Library (Ohio)
  • Elba Public Library (Alabama)
  • Essex Library Association (Connecticut)
  • Everett Public Library (Washington)
  • Feleti Barstow Public Library (American Samoa)
  • Forsyth County Public Library (North Carolina)
  • Hartford History Center, Hartford Public Library (Connecticut)
  • Heritage Public Library (Virginia)
  • Huntsville-Madison County Public Library (Alabama)
  • James Blackstone Memorial Library (Connecticut)
  • Jefferson Parish Library (Louisiana)
  • Jefferson-Madison Regional Library (Virginia)
  • Laramie County Library System (Wyoming)
  • Lawrence Public Library (Massachusetts)
  • Los Angeles Public Library (California)
  • Mill Valley Public Library, Lucretia Little History Room (California)
  • Missoula Public Library (Montana)
  • Niagara Falls Public Library (New York)
  • Pueblo City-County Library District (Colorado)
  • Rochester Public Library (New York)
  • Santa Cruz Public Libraries (California)
  • South Pasadena Public Library (California)
  • State Library of Pennsylvania (Pennsylvania)
  • Tangipahoa Parish Library (Louisiana)
  • The African American Research Library and Cultural Center (Florida)
  • The Ferguson Library (Connecticut)
  • Three Rivers Public Library District (Illinois)
  • Virginia Beach Public Library (Virginia)
  • Waltham Public Library (Massachusetts)
  • Watsonville Public Library (California)
  • West Virginia Library Commission (West Virginia)
  • William B Harlan Memorial Library (Kentucky)
  • Worcester Public Library (Massachusetts)
  • Your Heritage Matters (North Carolina)

Seeking Public Library Participants for Community History Web Archiving Program

Local history collections are necessary to understanding the life and culture of a community. As methods for sharing  information have shifted towards the web, there are many more avenues for community members to document diverse experiences.  Public libraries play a critical role in building community-oriented archives and these collections  are particularly important in recording the impact of unprecedented events on the lives of local citizens. 

Last week, we announced a major national expansion of our Community Webs program providing infrastructure, services, and training to public librarians to archive local history as documented on the web… We now invite public libraries in the United States and cultural heritage organizations in U.S. territories to apply to join the Community Webs program. Participants in the program receive free web archiving and technical services, education, professional development, and funding to build  community history web archives, especially collections documenting the lives of patrons and communities traditionally under-represented in the historical record.

If you are a public librarian interested in joining the Community Webs program please review the full call for applications and the program FAQs. Online applications are being accepted through Sunday, January 31, 2021

“Whether documenting the indie music scene of the 1990s, researching the history of local abolitionists and formerly enslaved peoples, or helping patrons research the early LGBT movement, I am frequently reminded of what was not saved or is not physically present in our collections. These gaps or silences often reflect subcultures in our community.” – Dylan Gaffney, Forbes Library, in Northampton, MA

The program is seeking public libraries to join a diverse network of 150+ organizations  that are:

  • Documenting local history by saving web-published sites, stories and community engagement on the web.
  • Growing their professional skills and increasing institutional technical capacity by engaging in a supportive network of peer organizations pursuing this work.
  • Building a public understanding of web archiving as a practice and its importance to preserving 21st century community history and underrepresented voices.

Current Community Webs cohort members have created nearly 300 publicly available local history web archive collections on topics ranging from COVID-19, to local arts and culture, to 2020 local and U.S. elections. Collecting the web-published materials of local organizations, movements and individuals is often the primary way to document their presence for future historians.

“During the summer of 2016, Baton Rouge witnessed the shooting of Alton Sterling, the mass shooting of Baton Rouge law enforcement, and the Great Flood of 2016. While watching these events unfold from our smartphones and computers, we at the East Baton Rouge Parish Library realized this information might be in jeopardy of never being acquired and preserved due to a shift in the way information is being created and disseminated.” – Emily Ward, East Baton Rouge Parish Library

Benefits of participation in Community Webs include:

  • A three-year subscription to the Archive-It web archiving service.
  • Funding to support travel to a full-day Community Webs National Symposium (projected for 2021 and in 2022) and other professional development opportunities. 
  • Extensive training and educational resources provided by professional staff.
  • Membership in an active and diverse community of public librarians across the country. 
  • Options to increase access (and discoverability) to program collections via hubs, such as DPLA.
  • Funding to support local outreach, public programming, and community collaborations. 

Please feel free to email us with any questions and be sure to apply by Sunday, January 31, 2021.

Community Webs Program Receives $1,130,000 Andrew W. Mellon Foundation Award for a National Network of Public Libraries Building Local History Web Archives

More than ever, the lives of communities are documented online. The web remains a vital resource for traditionally under-represented groups to write and share about their lives and experiences. Preserving this web-published material, in turn, allows libraries to build more expansive, inclusive, and community-oriented archival collections.

In 2017, the Internet Archive’s Archive-It service launched the program, “Community Webs: Empowering Public Libraries to Create Community History Web Archives.” The program provides training, professional development, cohort building, and technical services for public librarians to curate community archives of websites, social media, and online material documenting the experiences of their patrons, especially those often underrepresented in traditional physical archives. Since its launch, the program has grown to include 40 public libraries in 21 states that have built almost 300 collections documenting local civic life, especially of marginalized groups, creating an archive totaling over 50 terabytes and tens of millions of individual digital documents, images, audio-video, and more. The program received additional funding in 2019 to continue its work and focus on strategic planning, partnering with the Educopia Institute to ensure the growth and sustainability of the program and the cohort.

We are excited to announce that Community Webs has received $1,130,000 in funding from The Andrew W. Mellon Foundation for “Community Webs: A National Network of Public Library Web Archives Documenting Local History & Underrepresented Groups,” an nationwide expansion of the program to include a minimum of 2 public libraries in each of the 50 United States, plus additional local history organizations in U.S territories, for a total of 150-200 participating public libraries and heritage organizations. Participants will receive web archiving and access services, training and education, and funds to promote and pursue their community archiving. The Community Webs National Network will also make the resulting public library local history community web archives available to scholars through specialized access tools and datasets, partner with affiliated national discovery and digital collections platforms such as DPLA, and build partnerships and collaborations with state and regional groups advancing local history digital preservation efforts. We thank The Andrew W. Mellon Foundation for their generous support to grow this program nationwide and empower hundreds of public librarians to build archives that elevate the voices, lives, and events of their underrepresented communities and ensure this material is permanently available to patrons, students, scholars, and citizens.

Over the course of the Community Webs program, participating public libraries have created diverse collections on a wide range of topics, often in collaboration with members of their local communities. Examples include:

  • Community Webs members have created collections related to the COVID-19 pandemic, including Schomburg Center for Research in Black Culture’s “Novel Coronavirus COVID-19” collection which focuses on “the African diasporan experiences of COVID-19 including racial disparities in health outcomes and access, the impact on Black-owned businesses, and cultural production.” Athens Regional Library System created a collection of “Athens, Georgia Area COVID-19 Response” which focuses on the social, economic and health impacts of COVID-19 on the local community, with specific attention on community efforts to support frontline workers. A recent American Libraries article featured the COVID archiving work of public libraries.
  • Columbus Metropolitan Library’s archive of “Immigrant Experience”, a collection of websites on the activities, needs, and culture of immigrant communities in Central Ohio.
  • Sonoma County Public Library’s “North Bay Fires, 2017” collection documenting when “devastating firestorms swept through Sonoma, Napa, and Mendocino Counties” and part of their “Sonoma Responds: Community Memory Archive.”
  • Birmingham Public Library’s “LGBTQ in Alabama” collection “documenting the history and experiences of the LGBTQ community in Alabama.”
Community Webs public librarians at IA HQ

We look forward to expanding the Community Webs program nationwide in order to enable hundreds of public libraries to continue to build web collections documenting their communities, especially in these historic times.

We expect to put out a Call for Applications in early December for public libraries to join Community Webs. Please pass along this opportunity to your local public library. For more information on the program, check out our website or email us with questions.

Internet Archive Participates in DOAJ-Led Collaboration to Improve the Preservation of OA Journals

Since 2017, Internet Archive has pursued dedicated technical and partnership work to help preserve and provide perpetual access to open access scholarly literature and other outputs. See our original announcement related to this work and a recent update on progress. The below official press release announces an exciting new multi-institutional collaboration in this area.

The Directory of Open Access Journals (DOAJ), the CLOCKSS Archive, Internet Archive, Keepers Registry/ISSN International Centre and Public Knowledge Project (PKP) have agreed to partner to provide an alternative pathway for the preservation of small-scale, APC-free, Open Access journals.

The recent study authored by M.Laakso, L.Matthias, and N.Jahn has revived academia’s concern over the disappearance of the scholarly record disseminated in Open Access (OA) journals.

Their research focuses on OA journals as at risk of vanishing, and “especially small-scale and APC-free journals […] with limited financial resources” that often “opt for lightweight technical solutions” and “cannot afford to enroll in preservation schemes.” The authors have used data available in the Directory of Open Access Journals to come up with the conclusion that just under half of the journals indexed in DOAJ participate in preservation schemes. Their findings “suggest that current approaches to digital preservation are successful in archiving content from larger journals and established publishing houses but leave behind those that are more at risk.” They call for new preservation initiatives “to develop alternative pathways […] better suited for smaller journals that operate without the support of large, professional publishers.”

Answering that call, the joint initiative proposed by the five organisations aims at offering an affordable archiving option to OA journals with no author fees (“diamond” OA) registered with DOAJ, as well as raising awareness among the editors and publishers of these journals about the importance of enrolling with a preservation solution. DOAJ will act as a single interface with CLOCKSS, PKP and Internet Archive and facilitate a connection to these services for interested journals. Lars Bjørnhauge, DOAJ Managing Editor, said: “That this group of organisations are coming together to find a solution to the problem of “vanishing” journals is exciting. It comes as no surprise that journals with little to no funding are prone to disappearing. I am confident that we can make a real difference here.”

Reports regarding the effective preservation of the journals’ content will be aggregated by the ISSN International Centre (ISSN IC) and published in the Keepers Registry. Gaëlle Béquet, ISSN IC Director, commented: “As the operator of the Keepers Registry service, the ISSN International Centre receives inquiries from journal publishers looking for archiving solutions. This project is a new step in the development of our service to meet this need in a transparent and diverse way involving all our partners.”

About 50% of the journals identified by DOAJ as having no archiving solution in place use the Open Journal System (OJS). Therefore, the initiative will also identify and encourage journals on PKP’s OJS platform to preserve their content in the PKP Preservation Network (PKP PN), or to use another supported solution if the OJS instance isn’t new enough to be compatible with the PN integration (OJS 3.1.2+). 

The partners will then follow up by assessing the success and viability of the initiative with an aim to open it up to new archiving agencies and other groups of journals indexed in DOAJ to consolidate preservation actions and ensure service diversity.

DOAJ will act as the central hub where publishers will indicate that they want to participate. Archiving services, provided by CLOCKSS, Internet Archive and PKP will expand their existing capacities. These agencies will report their metadata to the Keepers Registry to provide an overview of the archiving efforts. 

Project partners are currently exploring business and financial sustainability models and outlining areas for technical collaboration.


DOAJ is a community-curated list of peer-reviewed, open access journals and aims to be the starting point for all information searches for quality, peer reviewed open access material. DOAJ’s mission is to increase the visibility, accessibility, reputation, usage and impact of quality, peer-reviewed, open access scholarly research journals globally, regardless of discipline, geography or language. DOAJ will work with editors, publishers and journal owners to help them understand the value of best practice publishing and standards and apply those to their own operations. DOAJ is committed to being 100% independent and maintaining all of its services and metadata as free to use or reuse for everyone.

CLOCKSS is a not-for-profit joint venture among the world’s leading academic publishers and research libraries whose mission is to build a sustainable, international, and geographically distributed dark archive with which to ensure the long-term survival of Web-based scholarly publications for the benefit of the greater global research community. https://www.clockss.org.

Internet Archive is a non-profit digital library, top 200 website at https://archive.org/, and archive of over 60PB of millions of free books, movies, software, music, websites, and more. The Internet Archive partners with over 800 libraries, universities, governments, non-profits, scholarly communications, and open knowledge organizations around the world to advance the shared goal of “Universal Access to All Knowledge.” Since 2017, Internet Archive has pursued partnerships and technical work with a focus on preserving all publicly accessible research outputs, especially at-risk, open access journal literature and data, and providing mission-aligned, non-commercial open infrastructure for the preservation of scholarly knowledge.

Keepers Registry hosted by the ISSN International Centre, an intergovernmental organisation under the auspices of UNESCO, is a global service that monitors the archiving arrangements for continuing resources including e-serials. A dozen archiving agencies all around the world currently report to Keepers Registry. The Registry has three main purposes: 1/ to enable librarians, publishers and policy makers to find out who is looking after what e-content, how, and with what terms of access; 2/ to highlight e-journals which are still “at risk of loss” and need to be archived; 3/ to showcase the archiving organizations around the world, i.e. the Keepers, which provide the digital shelves for access to content over the long term.

PKP is a multi-university and long-standing research project that develops (free) open source software to improve the quality and reach of scholarly publishing. For more than twenty years, PKP has played an important role in championing open access. Open Journal Systems (OJS) was released in 2002 to help reduce cost as a barrier to creating and consuming scholarship online. Today, it is the world’s most widely used open source platform for journal publishing: approximately 42% of the journals in the DOAJ identify OJS as their platform/host/aggregator. In 2014, PKP launched its own Private LOCKSS Network (now the PKP PN) to offer OJS journals unable to invest in digital preservation a free, open, and trustworthy service. 

For more information, contact: 

DOAJ: Dom Mitchell, dom@doaj.org

CLOCKSS: Craig Van Dyck, cvandyck@clockss.org

Internet Archive: Jefferson Bailey, jefferson@archive.org

Keepers Registry: Gaëlle Béquet, gaelle.bequet@issn.org

PKP: James MacGregor, jbm9@sfu.ca

Archive-It and Archives Unleashed Join Forces to Scale Research Use of Web Archives

Archived web data and collections are increasingly important to scholarly practice, especially to those scholars interested in data mining and computational approaches to analyzing large sets of data, text, and records from the web. For over a decade Internet Archive has worked to support computational use of its web collections through a variety of services, from making raw crawl data available to researchers, performing customized extraction and analytic services supporting network or language analysis, to hosting web data hackathons and having dataset download features in our popular suite of web archiving services in Archive-It. Since 2016, we have also collaborated with the Archives Unleashed project to support their efforts to build tools, platforms, and learning materials for social science and humanities scholars to study web collections, including those curated by the 700+ institutions using Archive-It

We are excited to announce a significant expansion of our partnership. With a generous award of $800,000 (USD) to the University of Waterloo from The Andrew W. Mellon Foundation, Archives Unleashed and Archive-It will broaden our collaboration and further integrate our services to provide easy-to-use, scalable tools to scholars, researchers, librarians, and archivists studying and stewarding web archives.  Further integration of Archives Unleashed and Archive-It’s Research Services (and IA’s Web & Data Services more broadly) will simplify the ability of scholars to analyze archived web data and give digital archivists and librarians expanded tools for making their collections available as data, as pre-packaged datasets, and as archives that can be analyzed computationally. It will also offer researchers a best-of-class, end-to-end service for collecting, preserving, and analyzing web-published materials.

The Archives Unleashed team brings together a team of co-investigators.  Professor Ian Milligan, from the University of Waterloo’s Department of History, Jimmy Lin, Professor and Cheriton Chair at Waterloo’s Cheriton School of Computer Science, and Nick Ruest, Digital Assets Librarian in the Digital Scholarship Infrastructure department of York University Libraries, along with Jefferson Bailey, Director of Web Archiving & Data Services at the Internet Archive, will all serve as co-Principal Investigators on the “Integrating Archives Unleashed Cloud with Archive-It” project. This project represents a follow-on to the Archives Unleashed project that began in 2017, also funded by The Andrew W. Mellon Foundation.

“Our first stage of the Archives Unleashed Project,” explains Professor Milligan, “built a stand-alone service that turns web archive data into a format that scholars could easily use. We developed several tools, methods and cloud-based platforms that allow researchers to download a large web archive from which they can analyze all sorts of information, from text and network data to statistical information. The next logical step is to integrate our service with the Internet Archive, which will allow a scholar to run the full cycle of collecting and analyzing web archival content through one portal.”

“Researchers, from both the sciences and the humanities, are finally starting to realize the massive trove of archived web materials that can support a wide variety of computational research,” said Bailey. “We are excited to scale up our collaboration with Archives Unleashed to make the petabytes of web and data archives collected by Archive-It partners and other web archiving institutions around the world more useful for scholarly analysis.” 

The project begins in July 2020 and will begin releasing public datasets as part of the integration later in the year. Upcoming and future work includes technical integration of Archives Unleashed and Archive-It, creation and release of new open-source tools, datasets, and code notebooks, and a series of in-person “datathons” supporting a cohort of scholars using archived web data and collections in their data-driven research and analysis. We are grateful to The Andrew W. Mellon Foundation for their support of this integration and collaboration in support of critical infrastructure supporting computational scholarship and its use of the archived web.

Primary contacts:
IA – Jefferson Bailey, Director of Web Archiving & Data Services, jefferson [at] archive.org
AU – Ian Milligan, Professor of History, University of Waterloo, i2milligan [at] uwaterloo.ca

“Community Webs” Receives Additional Funding to Further Public Library Local History Web Collecting

In 2017, our Archive-It service was awarded funding from the Institute of Museum and Library Services (IMLS) for the 2-year project “Community Webs: Empowering Public Librarians to Create Community History Web Archives.” The program has been providing training and technical infrastructure for a diverse group of librarians nationwide to develop expertise in creating collections of historically valuable web-published materials documenting their local communities and under-represented communities. In response to an unexpectedly large group of applicants, and with additional internal funding, we were able to expand the cohort to a total of 28 libraries from 16 states. The launch announcement and the dedicated website have further information about the program and its progress.

We are excited to announce that IMLS has recently provided additional supplementary funding to Community Webs! The additional funding will allow us to focus on program evaluation, expansion, and strategic planning. We are very pleased to be working with the Educopia Institute in support of this work and will benefit from their vast expertise in community cultivation and program facilitation.

Over the course of the original 2-year Community Webs program, the 28 participating libraries created hundreds of archived collections totaling more than 40 terabytes of data, gave dozens of professional presentations at local and national conferences, held many public programs and patron-facing events, and attended numerous meet-ups and cohort events. As well, the program created a suite of open educational resources, online courses, and other training materials supporting digital curation skills development, local history web collecting, and community formation. Some sample collections created as part of the program include:

#HashtagSyllabusMovement by Schomburg Center for Research in Black Culture
LGBTQ in Alabama by Birmingham Public Library
D.C. Punk (Web) Archive by DC Public Library, Special Collections
North Bay Fires, 2017 by Sonoma County Public Library
Food Culture by Athens (GA) Regional Library System
Movimiento Cosecha Grand Rapids by Grand Rapids Public Library

The program’s website has links to each participating institution’s collections page.

We are grateful to IMLS for the additional funding to continue this popular program, excited to work with Educopia on further community development, and encourage any public libraries interested in participating to contact us.

Archiving Information on the Novel Coronavirus (Covid-19)

The Internet Archive’s Archive-It service is collaborating with the International Internet Preservation Consortium’s (IIPC) Content Development Group (CDG) to archive web-published resources related to the ongoing Novel Coronavirus (Covid-19) outbreak. The IIPC Content Development Group consists of curators and professionals from dozens of libraries and archives from around the world that are preserving and providing access to the archived web. The Internet Archive is a co-founder and longtime member of the IIPC. The project will include both subject-expert curation by IIPC members as well as the inclusion of websites nominated by the public (see the nomination form link below).

Due to the urgency of the outbreak, archiving of nominated web content will commence immediately and continue as needed depending on the course of the outbreak and its containment. Web content from all countries and in any language is in scope. Possible topics to guide nominations and collections: 

  • Coronavirus origins 
  • Information about the spread of infection 
  • Regional or local containment efforts
  • Medical/Scientific aspects
  • Social aspects
  • Economic aspects
  • Political aspects

Members of the general public are welcomed to nominate websites and web-published materials using the following web form: https://forms.gle/iAdvSyh6hyvv1wvx9. Archived information will also be available soon via the IIPC’s public collections in Archive-It. [March 23, 2020 edit: the public collection can now be found here, https://archive-it.org/collections/13529.]

Members of the general public can also take advantage of the ability to upload non-web digital resources directly to specific Internet Archive collections such as Community Video or Community Texts. For instance, see this collection of “Files pertaining to the 2019–20 Wuhan, China Coronavirus outbreak.” We recommend using a common subject tag, like coronavirus to facilitate search and discovery. Fore more information on uploading materials to archive.org, see the Internet Archive Help Center.

A special thanks to Alex Thurman of Columbia University and Nicola Bingham of the British Library, the co-chairs of the IIPC CDG, and to other IIPC members participating in the project. Thanks as well to any and all public nominators assisting with identifying and archiving records about this significant global event.

Archiving Online Local News with the News Measures Research Project

Over the past two years Archive-It, Internet Archive’s web archiving service, has partnered with researchers at the Hubbard School of Journalism and Mass Communication at University of Minnesota and the Dewitt Wallace Center for Media and Democracy at Duke University in a project designed to evaluate the health of local media ecosystems as part of the News Measures Research Project, funded by the Democracy Fund. The project is led by Phil Napoli at Duke University and Matthew Weber at University of Minnesota. Project staff worked with Archive-It to crawl and archive the homepages of 663 local news websites representing 100 communities across the United States. Seven crawls were run on single days from July through September and captured over 2.2TB of unique data and 16 million URLs. Initial findings from the research detail how local communities cover core topics such as emergencies, politics and transportation. Additional findings look at the volume of local news produced by different media outlets, and show the importance of local newspapers in providing communities with relevant content. 

The goal of the News Measures Research Project is to examine the health of local community news by analyzing the amount and type of local news coverage in a sample of community. In order to generate a random and unbiased sample of communities, the team used US Census data. Prior research suggested that average income in a community is correlated with the amount of local news coverage; thus the team decided to focus on three different income brackets (high, medium and low) using the Census data to break up the communities into categories. Rural areas and major cities were eliminated from the sample in order to reduce the number of outliers; this left a list of 1,559 communities ranging in population from 20,000 to 300,000 and in average household income from $21,000 to $215,000. Next, a random sample of 100 communities was selected, and a rigorous search process was applied to build a list of 663 news outlets that cover local news in those communities (based on Web searches and established directories such as Cision).

The News Measures Research Project web captures provide a unique snapshot of local news in the United States. The work is focused on analyzing the nature of local news coverage at a local level, while also examining the broader nature of local community news. At the local level, the 100 community sample provides a way to look at the nature of local news coverage. Next, a team of coders analyzed content on the archived web pages to assess what is being covered by a given news outlet. Often, the websites that serve a local community are simply aggregating content from other outlets, rather than providing unique content. The research team was most interested in understanding the degree to which local news outlets are actually reporting on topics that are pertinent to a given community (e.g. local politics). At the global level, the team looked at interaction between community news websites (e.g. sharing of content) as well as automated measures of the amount of coverage.

The primary data for the researchers was the archived local community news data, but in addition, the team worked with census data to aggregate other measures such as circulation data for newspapers. These data allowed the team to examine the amount and type of local news changes depending on the characteristics of the community. Because the team was using multiple datasets, the Web data is just one part of the puzzle. The WAT data format proved particularly useful for the team in this regard. Using the WAT file format allowed the team to avoid digging deeply into the data – rather, the WAT data allowed the team to examine high level structure without needing to examine the content of each and every WARC record. Down the road, the WARC data allows for a deeper dive,  but the lighter metadata format of the WAT files has enabled early analysis.

Stay tuned for more updates as research utilizing this data continues! The websites selected will continue to be archived and much of the data are publicly available.