Competency E

Design, query, and evaluate information retrieval systems.

Section 1: Introduction

The design of an information retrieval system involves decisions about how information is organized for discovery, how users’ information needs should be translated into system inputs, and how system performance should be evluated. These processes are deeply interconnected: understanding how to design information retrieval systems can help you better query them effectively. Conversely, knowing how users actually query systems and what they consider a “good” result can inform better system design and evaluation metrics.

Design principles for a retrieval system begin with decisions about vocabulary control. Miller (2022) describes several distinct types of controlled vocabularies information professionals can choose from, including simple lists, synonym rings that group equivalent terms together, authority files that establish one preferred form for a name or concept, and thesauri, which map broader, narrower, and related relationships between terms. Tucker (2024) adds a further design distinction: whether a vocabulary’s terms are precoordinated, meaning a cataloger has already combined multiple concepts into a single heading before a document is indexed, like the Library of Congress Subject Headings’ multi-level structure, or postcoordinated, meaning a searcher combines separate terms at the time of search using tools like Boolean operators. Tucker notes that most vocabularies developed for computerized retrieval are postcoordinate, while many important legacy vocabularies, including LCSH, remain precoordinated. Tucker also names disambiguation as a core design concern: a given word could refer to multiple distinct concepts (i.e., “apple” could refer to the fruit or the technology company) and a good retrieval system will be able to direct the user towards the one that matches their intention.

Querying a system well requires more than knowing the correct syntax; it involves developing a feel for how a specific system’s tools interact. Brown (2021) frames effective searching as a skill built through deliberate technique rather than intuition alone, recommending strategies like considering synonyms before searching and adjusting a query iteratively based on initial results. This matches my own experience: since starting this program, I’ve become far more comfortable using tools like Boolean operators, proximity connectors, truncation, and parentheses, and, just as importantly, I’ve learned to treat an initial query as a first draft, which I may have to revise multiple times before feeling satisfied with the results I’ve retrieved.

Evaluating whether a system is serving user needs involves its own specific vocabulary. Tucker (2024) defines two key measures: recall, the ratio of relevant documents retrieved to all relevant documents that exist in the system, which answers how completely a search captured everything relevant; and precision, the ratio of retrieved documents that are actually relevant to the total number retrieved, which answers how much of what came back was actually useful. Critically, Tucker notes that these two measures trade off against each other: broadening a search to increase recall typically pulls in more irrelevant results and lowers precision, while narrowing a search to increase precision risks missing relevant material and lowers recall. Retrieving “more” results isn’t automatically better, and evaluating a system, or a specific search, involves deciding which of these two measures, recall or precision, matters more for the task at hand.

Throughout my coursework, I have discovered a real affinity for information retrieval systems, and learning the principles behind how they function has increased my appreciation for effective search interfaces. I expect to use this competency throughout my career because so much of information work involves building systems or navigating them on someone else’s behalf: designing how a collection’s materials will be described and found, helping a patron construct a search that actually returns the informationm they need, and knowing how to test if an information retrieval system, whether new or long-established, is adequately serving the people who rely on it.

References

Section 2: Evidence

My understanding of information retrieval system design, querying, and evaluation comes from coursework in database searching and metadata, spanning both hands-on system design and structured, comparative evaluation of professional retrieval platforms.

My first piece of evidence is a reflection on a group database design project from INFO 202, in which I authored specific field rules for a database cataloging chip products. My assigned role within the group was editor: I authored two of the group’s eight field rules, chip type and flavor, contributed to the statement of purpose, and edited the group’s rules for stylistic consistency after all members had submitted their contributions. This artifact demonstrates information retrieval system design principles directly, including some choices I only recognized as design failures in hindsight. My flavor rule shows a deliberate attempt to control for ambiguity: it used a multiselect controlled vocabulary with a defined fallback option, “Other/Not Specified,” for flavors outside the list, and explicitly restricted indexers to evaluating the package and product name rather than the ingredients list, to prevent inconsistent interpretation of what counted as a given flavor. My chip-type rule was less successful: it treated “corn” and “tortilla” as separate, mutually exclusive options, without ever defining within the rule itself how these categories differed, even though tortilla chips are typically made from corn.

During alpha testing, I also noticed a related problem in a rule I hadn’t authored: our calorie field’s ranges, labeled “low,” “medium,” and “high,” were too broad to be useful, since nearly every chip any of us tested fell into the “medium” bucket. I recognized this as a design flaw at the time but did not raise it with the group before we finalized the database, partly because everyone wanted to finish the assignment. This artifact demonstrates that good database design is not just about writing clear rules but also anticipating how real users and indexers will interpret a field’s boundaries, understanding the underlying range of values existing in the items being described, and deciding how much ambiguity is allowable in a field before it becomes unusable for consistent data entry. Poor rule specification produces the kind of inconsistent, hard-to-search data that later undermines a database’s usefulness, regardless of how well the query interface itself is built.

My second piece of evidence is a quiz from INFO 244, testing search construction skills in Dialog using Boolean operators, proximity connectors, truncation, and parentheses. In question 3, I constructed a search on students’ interactions with therapy animals reducing anxiety, using all of these tools together in the correct order of operations, then tested and refined the query after reviewing my results: my initial search returned false positives from unrelated topics like yoga poses (“downward dog,” “cat” pose) and art therapy (abbreviated “CAT”), which led me to narrow my proximity connectors and remove a term that wasn’t adding useful results. Other questions on the quiz demonstrate specific query principles I have learned, including the precision/recall tradeoff between Dialog’s two proximity connectors, PRE/ (higher precision, requiring one term to precede the other) and NEAR/ (higher recall, matching either order), and the advantages and disadvantages of cross-file searching, which increases recall by searching multiple databases at once but sacrifices access to each database’s specific controlled vocabulary and limiters. By working through search problems in multiple databases, I have learned that effective querying involves iteratively evaluating a query’s results and adjusting specific tools, like proximity distance, term choice, and operator selection, in response.

My third piece of evidence is a second quiz from INFO 244 focused on the database Web of Science, in which I answered several questions about how to best use this powerful resource. For question 8, I evaluated how the same scholarly article is indexed differently across Web of Science and Dialog’s Social SciSearch database. I found that both systems assigned overlapping subject terms, including digital literacy and AI, but Dialog’s indexing, generated by human indexers, also included broader disciplinary terms like computer science, cybernetics, and ergonomics directly within the same subject field, while Web of Science split similar disciplinary classification into a separate, algorithmically generated field distinct from its topical Keywords Plus. I concluded that a searcher using Web of Science would need to strategically combine terms from both of these separate fields to replicate the fuller picture Dialog’s single, human-curated field provided on its own. Evaluating a retrieval system requires understanding how its underlying indexing methodology, whether human-curated or algorithmically generated, shapes what a searcher needs to do differently to retrieve equivalent information across systems.

My fourth piece of evidence is a comparative research paper from INFO 256, in which I evaluated six different archival search interfaces by conducting the same search query in each one. This artifact demonstrates my ability to evaluate differences in query language and system design across comparable retrieval systems: one platform’s lack of Boolean search capability meant I had to run entirely separate searches for synonymous terms rather than combining them into a single query, while another platform’s specific date-range search returned an unmanageably large result set until I learned to combine it with more specific subject-heading filters. I also evaluated differences in each system’s underlying organization, noting that platforms differed in whether they supported controlled vocabulary lookup, meaning that searchers unfamiliar with an archive’s specific indexing terms could easily miss relevant materials in some systems but not others. Together, these two evaluation pieces demonstrate that assessing an information retrieval system requires examining both its indexing methodology and its query functionality, since a system can fail a searcher in either dimension regardless of how well the other is designed.

Section 3: Conclusion

Designing, querying, and evaluating an information retrieval system turn out to be harder to separate in practice than the competency’s three verbs might suggest: design decisions like database rules shape what searchers can later find, and evaluating professional systems side by side taught me much about querying strategy. In my future career, I expect to apply this competency directly to cataloging and metadata work, building systems with searchers’ actual needs and vocabulary in mind, and staying alert to the precision/recall tradeoffs any design choice creates. To stay current in this area, I plan to consult the Association for Information Science and Technology’s (ASIS&T) Digital Library, particularly the Journal of the Association for Information Science and Technology (JASIST), as a source for ongoing research in information retrieval, information behavior, and information system design. I also plan to follow the National Information Standards Organization’s (NISO) work on information discovery and indexing, particularly its Information Discovery & Interchange Topic Committee. I can also use NISO’s Criteria for Indexes standard as a reference when I encounter questions about the design and organization of indexes. These resources will help me continue connecting the technical principles of retrieval system design with the practical concerns of library users: how information is described, how searchers formulate queries, and how effectively systems help them find what they need.