Greg Leppert

Greg Leppert

@leppert.me

Working on AI and access to knowledge at Harvard. Executive Director of the Institutional Data Initiative; Chief Technologist of the Berkman Klein Center.

1.1K followers141 following46 posts

The Institutional Data Initiative continues our work to increase the computational accessibility of the ~1M volumes in Insitutional Books. Today’s releases add access to 22M photos and illustrations as well as 1.9B annotated paragraphs. Dig in.

IDI@institutional.org

Institutional Books is expanding. The Institutional Data Initiative @institutional.org at @harvardlawadmin.bsky.social Library has added paragraph-level text analysis and 22M extracted images to our work on ∼1M books, opening new possibilities for research, experimentation, and discovery. 🧵

Immensely proud of this collaboration with @bpl.boston.gov. A dataset of 1.47M newspaper pages and a new processing pipeline that breaks pages down to their component parts, increasing OCR accuracy and computational addressability. The historical record has never been more searchable.

IDI@institutional.org

When a newspaper becomes a wall of OCR text, columns collapse, headlines mix with ads, and reading order disappears. The Institutional Data Initiative (@institutional.org) and Boston Public Library (@bpl.boston.gov) built a pipeline and dataset to make historical newspapers more accessible.🧵

This project is the first of many that @institutional.org is undertaking to help libraries and other knowledge institutions enable computational access to their collections. If you represent a library and would like to work together, we'd love to hear from you.

Organizational decision-making is fertile ground for AI-assisted research, but board minutes often bury key decisions in narrative prose. Using reasoning models, we worked with U-M Library to extract & structure these decisions so researchers can use computational tools to study the evolution of U-M

Across 186yrs, we identified leadership changes, fundraising progress, and other strategic plans including the creation of race and gender policies following the Civil Rights Act as well as a 1921 $1M drive for a “women’s building,” launched after women were barred from entering the Michigan Union.

When libraries join Google Books, Google not only scans their books, it also makes a wealth of image, OCR, & metadata available to them via the Google Return Interface (GRIN). But working with GRIN can be challenging, so we're releasing a tool to make it easier. www.institutional.org/posts/grin-t...

institutional.orgAnnouncing the release of GRIN TransferGRIN Transfer, an open source tool that allows Google Books partner libraries to more easily access their Google Books collection.
View full thread

When libraries join Google Books, Google not only scans their books, it also makes a wealth of image, OCR, & metadata available to them via the Google Return Interface (GRIN). But working with GRIN can be challenging, so we're releasing a tool to make it easier. www.institutional.org/posts/grin-t...

institutional.orgAnnouncing the release of GRIN TransferGRIN Transfer, an open source tool that allows Google Books partner libraries to more easily access their Google Books collection.

We learned this lesson over the months it took to download 1M of Harvard Library's books for our Institutional Books release. As a result, many libraries have yet to take full advantage of the wonderful resources GRIN provides.

That's why we built GRIN Transfer: a tool for downloading collections, big or small. GRIN Transfer handles request batching, failure recovery, and data aggregation so that libraries can focus on using the data rather than simply gaining access to it. www.institutional.org/posts/grin-t...

institutional.orgAnnouncing the release of GRIN TransferGRIN Transfer, an open source tool that allows Google Books partner libraries to more easily access their Google Books collection.

Greg Leppert reposted

This starts in an hour.

Greg Leppert@leppert.me

This Monday, @institutionaldatainitiative.org will host Petr Knoth to share his experience leading CORE ("The world’s largest collection of open access research papers") as the rise of AI brings new meaning, and challenges, to stewarding knowledge repositories. Join us virtually via the link below.

The @institutionaldatainitiative.org is proud to support The New Commons challenge. $100k grants along with mentorship. Let's get impactful data into the AI ecosystem.

The GovLab@thegovlab.org

(1/4) CALL FOR APPLICATIONS FOR DATA COMMONS FOR AI 🏆Today, The Open Data Policy Lab (a collaboration btwn The GovLab & @microsoft.com launched The New Commons Challenge—an innovation challenge to foster the creation of data commons that can support generative AI developed in the public interest.

Greg Leppert reposted

To start the weekend, we've got a brand new experience for case law on CourtListener. It has better typography, more features and metadata, five million scanned decisions from @harvardlil.bsky.social, and a lot more. Read all about it and let us know what you think: free.law/2025/03/21/c...

free.lawA Faster, Smarter, Unified Case Law ExperienceA redesigned case law modernizes the reading experience with enhanced layout and typography, more advanced features, better speed, and more.

Our goal is to develop methods and tools that can support expert staff at libraries everywhere, increasing the breadth of materials that can be digitized and the speed at which they’re made accessible to the public. Learn more at BPL: www.bpl.org/news/boston-...

bpl.orgBoston Public Library Expands Access to Collections Through AI-Enhanced DigitizationBOSTON, MA – March 12, 2025 - The Boston Public Library (BPL) is launching a large-scale digitization project to unlock hundreds of thousands…

IDI and BPL are working to change this by collaborating at the outset of a large digitization project, exploring how AI might complement human expertise and strengthen the process in its earliest stages.

Together, we’ll research opportunities to generate machine-readable representations of items, add searchable metadata, and begin the structuring of entire collections—all at the moment each item leaves the imaging station.

BPL is embarking on a new initiative to digitize hundreds of thousands of historic items. Conventional approaches to this scale lead to an impossible choice: sacrifice depth for breadth or drastically limit what gets digitized. AI tools can help, but they’re relegated to the end of the process.

If you're part of a library, university, or other knowledge institution and interested in working with a team of data scientists to refine and publish your data, we'd love to chat. And if you're a data scientist or community builder interested in working with institutions, we're hiring.

With our digitization at Harvard Law School Library, we'll work to increase access to unique collections, such as the Supreme Court Records and Briefs that are critical to understanding decision-making at the highest U.S. court yet remain largely inaccessible.

IDI is building a collection of large, impactful, and widely available datasets to increase AI’s accessibility and diversity while reaffirming institutions as stewards of knowledge.

The key is involving the institutions themselves in the conversation. Their missions are as much a reflection of the cultures they help to preserve as the data itself, and integrating them is critical as we look for new models to foster and interact with knowledge.

Yesterday we launched the Institutional Data Initiative at Harvard Law School Library to work with libraries, government agencies, and other knowledge institutions to help refine and publish their collections as data, with an eye toward AI. 🧵 bsky.app/profile/inst...

View full thread

Our launch is generously supported by @microsoft.com and OpenAI, and these projects are just the start. If you’re part of an institution, we’d love to hear how we can help. If you’re an AI/ML researcher, we'd love to collaborate. We're also hiring for our core team, here at Harvard. Reach out.

Yesterday we launched the Institutional Data Initiative at Harvard Law School Library to work with libraries, government agencies, and other knowledge institutions to help refine and publish their collections as data, with an eye toward AI. 🧵 bsky.app/profile/inst...

View full thread

Our first project is a collection of 1M public domain books, scanned at Harvard Library as part of the Google Books project. We're fortunate enough to have the support of Google in releasing this dataset and will do so in early 2025 with accompanying analysis.

We're also working with Boston Public Library on an active scanning project of theirs involving millions of pages taken from public domain newspapers. We're hoping to make progress on some of the classic OCR challenges presented by newspapers, while evaluating the resulting data for model training.

Yesterday we launched the Institutional Data Initiative at Harvard Law School Library to work with libraries, government agencies, and other knowledge institutions to help refine and publish their collections as data, with an eye toward AI. 🧵 bsky.app/profile/inst...

View full thread

Our goal is to work within the missions of institutions to help unlock and improve access to their collections, not only for AI uses but traditional patron access as well. This work can help expand the breadth of people that institutional collections and AI models are able to empower.

Yesterday we launched the Institutional Data Initiative at Harvard Law School Library to work with libraries, government agencies, and other knowledge institutions to help refine and publish their collections as data, with an eye toward AI. 🧵 bsky.app/profile/inst...

Data access plays an important role in the development of AI models. It helps define who is represented in them, who they empower, and even who can build them. Data integrity, as a result, is important as well.

Data access and integrity are two things institutions know well. They hold vast collections and think a lot about knowledge access and stewardship. Stewardship is a nice word because it conveys a sense of integrity over time, and time is something institutions excel at utilizing.

You are up to date.