{"id":18812,"date":"2025-09-29T14:00:52","date_gmt":"2025-09-29T14:00:52","guid":{"rendered":"https:\/\/library.udel.edu\/news\/?p=18812"},"modified":"2025-11-03T16:06:07","modified_gmt":"2025-11-03T16:06:07","slug":"ai-bots-swarm-udspace","status":"publish","type":"post","link":"https:\/\/library.udel.edu\/news\/2025\/09\/29\/ai-bots-swarm-udspace\/","title":{"rendered":"AI bots swarm UDSpace for information"},"content":{"rendered":"

Article by Jamie Washington<\/span>\u00a0Photo illustration by Jeffrey C. Chase<\/span><\/em><\/p>\n

UD Library\u2019s IT stops AI bots from making UDSpace inaccessible<\/h2>\n
\n

Long gone are the days of slowly sifting through dense pages of information for answers to mind-bending questions or remaining up-to-date with scientific discoveries. Now, artificial-intelligence bots are doing the sorting and summarizing for us by swarming scholarly content as if it were mounds of honey.<\/p>\n

And\u00a0UDSpace<\/a>, the University of Delaware\u2019s digital repository, is dripping with honey. This valuable resource is managed by UD\u2019s Library, Museum and Press, to preserve and provide long-term free and open access to scholarly research outputs produced by UD faculty, students and staff, which includes articles, white papers, reports, and theses and dissertations.<\/p>\n

In the last year, AI bots caused a significant spike in the volume of online information queries to UDSpace, which strained the capacity of the system and resulted in responses being delayed or not available at all. UDSpace was dealing with an unforeseen consequence of its own success: By making UD research freely available to anyone, it had actually made it less accessible to everyone.<\/p>\n

\u201cIt\u2019s all about the data,\u201d said Annie Johnson, associate university librarian for research, teaching and technology at the Library, Museum and Press. \u201cAI companies are constantly looking for more data to improve their Large Language Models (LLMs). UDSpace has massive amounts of high-quality data that is attractive to these companies.\u201d<\/p>\n

This phenomenon isn\u2019t unique to UDSpace \u2014 other institutional repositories are also dealing with this challenge, said Johnson. In the past year, AI bots have also made their presence known at\u00a0Duke University<\/a>\u00a0and the\u00a0University of North Carolina at Chapel Hill<\/a>, for example. Similar institutions throughout North America, Europe and other regions are experiencing the same challenges, according to a\u00a0recent survey<\/a>\u00a0by the GLAM-E Lab, a joint research center at the University of Exeter in England and the New York University School of Law that focuses on galleries, libraries, archives and museums.<\/p>\n

\u201cAt the start of a normal day, we\u2019re used to receiving less than 10 simultaneous queries a second to access information,\u201d said Mark Grabowski, head of Library Information Technology. \u201cAs you can imagine, we were surprised to have 80 to 100 simultaneous queries with users complaining about UDSpace being inaccessible.\u201d<\/p>\n

The Library\u2019s IT team diligently investigated the issue, finding that AI bots submitted multiple complex requests, including a researcher\u2019s name, a topic, date and title, in about half a second. Grabowski said that there are rules to AI bots making simple requests, \u201cbut the bots were mostly ignoring these rules, and the team noticed that many of the searches were randomly, and quickly, moving through UDSpace.\u201d<\/p>\n

With searches coming almost nonstop and from almost everywhere around the world, Colleen Estes, assistant head of Library IT, said that finding a resolution was a matter of trial and error. One of the initial ideas was to block IP addresses.<\/p>\n

\u201cOur goal is always to enable, improve and increase access to information,\u201d said Estes. \u201cWe knew that a permanent resolution couldn\u2019t be to block IP addresses because that would lead to us blocking entire countries, and we don\u2019t want to block real people from accessing information.\u201d<\/p>\n

Another temporary response was for the system to tell the AI bots that the information did not exist.<\/p>\n

\u201cWhen we did this, it was like poking a hornet\u2019s nest, and the bots tried harder,\u201d Grabowski said. \u201cThe unusually high 80-100 simultaneous searches jumped to 250 to 300 simultaneous searches.\u201d<\/p>\n

Since May 2025, UDSpace\u2019s message to the AI bots has changed: We have the information that you\u2019re looking for, but you\u2019re forbidden to access it.<\/p>\n

Grabowski said that UDSpace has three layers: the website that takes in a request, the application that processes the request and an underlying database that is queried for information based on the request. Now, there\u2019s a \u201cforbidden\u201d message that appears at the website level when the request comes from a bot, allowing the server to receive traffic, monitoring AI bots, but without requests getting to, and slowing down, the application.<\/p>\n

\u201cThere was one round of bot queries that ran for an entire weekend, determined to gather as much information as possible when no one was looking,\u201d he said. \u201cNow that we have \u2018forbidden\u2019 in place, they receive so many rejections, they stop making the request.\u201d<\/p>\n

AI is quickly changing reality, but human intelligence is still a potent force in solving challenges like AI bots.<\/p>\n

\u201cThis is all a testament to the ingenuity and the great lengths that our team has gone to to maintain access to information,\u201d Johnson said. \u201cAI is changing the rules, and this is an excellent example of UD Library\u2019s IT team quickly responding with an incredible solution that doesn\u2019t limit our students, faculty, staff, and the global community from accessing the information that UDSpace holds.\u201d<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Article by Jamie Washington\u00a0Photo illustration by Jeffrey C. Chase UD Library\u2019s IT stops AI bots from making UDSpace inaccessible Long gone are the days of slowly sifting through dense pages of information for answers to mind-bending questions or remaining up-to-date …<\/p>\n","protected":false},"author":180,"featured_media":18813,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[4],"tags":[],"class_list":["post-18812","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"acf":[],"publishpress_future_action":{"enabled":false,"date":"2026-04-21 06:07:06","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category"},"_links":{"self":[{"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/posts\/18812"}],"collection":[{"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/users\/180"}],"replies":[{"embeddable":true,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/comments?post=18812"}],"version-history":[{"count":2,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/posts\/18812\/revisions"}],"predecessor-version":[{"id":18861,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/posts\/18812\/revisions\/18861"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/media\/18813"}],"wp:attachment":[{"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/media?parent=18812"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/categories?post=18812"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/library.udel.edu\/news\/wp-json\/wp\/v2\/tags?post=18812"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}