Information Retrieval in Practice: A Python-Based Document Search Engine

This project is a PDF Search Engine that helps you quickly find relevant documents within a collection. It works by taking your natural language search query, intelligently processing it, and then presenting you with a ranked list of PDF files that best match your request, complete with content snippets and direct clickable links to the documents.

Visual Overview

Chapters

1. Streamlit User Interface
2. Search Engine Core Logic
3. Preprocessed Document Data
4. Text Preprocessing
5. TF-IDF Vectorization Model
6. Cosine Similarity Scoring
7. Document Path Resolver

Chapter 1: Streamlit User Interface

Imagine you've built a super-smart robot that can find exactly what you're looking for in a huge pile of documents. That's amazing! But how do you talk to this robot? How do you tell it what to search for, and how does it show you what it found?

This is where the Streamlit User Interface comes in. Think of it as the "face" of our PDF Search Engine. It's the friendly shop window and counter that helps you interact with our powerful search tool. Without it, our search engine would be like a brilliant mind with no way to communicate.

The problem this abstraction solves is making a complex search engine accessible and easy to use for anyone, even without knowing how the search works behind the scenes. It creates a beautiful, interactive web page right from our Python code!

Let's look at how a user would interact with our search engine using the Streamlit User Interface.

How You Use the Streamlit Interface

When you run our project, Streamlit automatically opens a web page in your browser. This web page is our Streamlit User Interface.

1. You open the app: You see a main title like "📄 PDF Search Engine" and a helpful "About" section on the side. This is all designed by Streamlit!

   This simple code snippet tells Streamlit to set up the page, display a main title, and some descriptive text. It's like putting up a sign outside our shop!

1. You see a sidebar: On the left, there's a panel with information about the app and some tips. This helps you understand what the app does.

   The st.sidebar. commands are used to place content specifically in the sidebar. This keeps the main search area clean.

2. You type your search query: There's a clear input box asking you to "Enter your search query:". You type what you're looking for there.

   st.text_input() creates a simple text box where users can type. Whatever they type gets stored in the user_query variable.

3. You get your search results: After you type your query and press Enter, the interface magically displays a list of search results. Each result shows the document name, a small snippet of its content, and a relevance score. They even look nice and organized because of some custom styling!

   Here, st.markdown() is used again, but this time it includes custom HTML code (<div>, <p>) to make each search result look like a neat little box. The f-string (the f before the quotes) helps us easily insert document information (doc['name'], doc['score'], etc.) into the display.

What Happens Under the Hood (Simplified)

When you interact with the Streamlit User Interface, it's like a well-coordinated dance between you and the application's logic.

Here's a simple sequence of events:

1. User Opens App: You navigate to the web address where the Streamlit app is running.
2. Initial Display: The src/index.py script runs from top to bottom. All the st.title, st.sidebar.markdown, and st.text_input commands are processed, creating the initial look of the web page. The query box is empty, so the "Enter a query above to search." message is shown.
3. User Enters Query: You type something in the search box.
4. Streamlit Re-runs: When you press Enter (or when the input changes), Streamlit is smart enough to realize something happened. It re-runs the entire src/index.py script.
5. Query Processing: This time, user_query is not empty. The code proceeds into the if user_query: block. The user_query is then sent to the Search Engine Core Logic (which we'll learn about in Chapter 2: Search Engine Core Logic) to find relevant documents.
6. Display Results: Once the Search Engine Core Logic returns the search results, the Streamlit User Interface takes these results and uses st.markdown() with custom styling to display them neatly on the page.

Styling Our Interface

You might have noticed that our search results look quite nice, with clear titles, snippets, and scores, all in little boxes. This is thanks to Custom CSS (Cascading Style Sheets). CSS is like the interior decorator for our web page; it tells the browser how elements should look (colors, fonts, spacing).

In src/index.py, we use st.markdown() with unsafe_allow_html=True to inject our own CSS rules:

This block of code is telling the browser:

• Any element with the class result-container should have a light grey background, some space inside (padding), space below it (margin-bottom), rounded corners (border-radius), and a light grey border.
• Any element with the class result-title should have a larger, bold font and a specific dark blue color.

This ensures that our search results are not just functional but also visually appealing and easy to read.

Conclusion

The Streamlit User Interface is our project's friendly face. It translates our complex Python code into an interactive web application that users can easily understand and navigate. It handles displaying titles, creating input fields, showing informative sidebars, and presenting search results in a clear, styled manner. It's the "shop window" that makes our powerful search engine accessible!

Now that we understand how users interact with our search engine, let's peek behind the curtain and dive into the "kitchen" – the brain of our application. In the next chapter, we'll explore the Search Engine Core Logic that powers the actual search process.

Chapter 2: Search Engine Core Logic

Chapter 2: Search Engine Core Logic

Welcome back! In Chapter 1: Streamlit User Interface, we learned about the "face" of our search engine – how you type your query and see the results. It's like the friendly shop assistant who takes your order and hands you the goods. But what happens behind the counter? How does the search engine actually find what you're looking for?

This is where the Search Engine Core Logic comes in. Think of it as the "brain" or the "master chef" of our entire PDF search application. When you type a query into the Streamlit interface, this core logic is the part that gets busy, performing all the complex steps needed to turn your words into meaningful search results.

The problem this abstraction solves is making sense of a user's request and finding the most relevant documents among potentially thousands of others. It's the central command center that makes our search engine smart and effective.

Let's imagine you type "machine learning algorithms" into our search engine. The Search Engine Core Logic would take this raw query and orchestrate a series of operations to find documents that match.

How the Search Engine Core Logic Works (The Big Picture)

At a high level, when you submit a query, the Search Engine Core Logic performs these key steps:

1. Understand Your Query: It first cleans up your query (e.g., removes punctuation, makes everything lowercase).
2. Translate Your Query: It converts your cleaned query into a special numerical format that computers can understand and compare.
3. Find Similar Documents: It compares this numerical query with the numerical representations of all the documents it knows about.
4. Rank the Best Matches: It sorts the documents from most relevant to least relevant based on how similar they are.
5. Prepare Results: Finally, it gets ready to show you the best documents, including their names, a little summary (snippet), and how relevant they are.

This entire sequence happens very quickly, giving you results almost instantly!

Using the Core Logic in Our App

You don't directly "call" the Search Engine Core Logic with a simple core_logic.search(query) command in src/index.py. Instead, src/index.py contains the sequence of steps that constitute the core logic. When user_query has content, our Streamlit app triggers these steps:

This sequence is the heart of our search engine. Let's break down each step.

Inside the Search Engine's Brain: A Step-by-Step Journey

Imagine your query "deep learning" starting its journey through the search engine's core logic:

Now, let's look at the actual code that performs these steps within src/index.py.

1. Preprocessing the User's Query

Before we can compare your query to documents, we need to make sure it's in a clean, consistent format. This is called Text Preprocessing. It's like polishing a rough gemstone to make it shine.

What's happening? If you type "What is Machine Learning?", this function would turn it into "what is machine learning". It removes capitalization, question marks, and any numbers, ensuring that "Machine" matches "machine" in our documents. We'll dive deeper into this in Chapter 4: Text Preprocessing.

2. Vectorizing the Query

Computers are great with numbers, but not so much with words directly. So, the next step is to convert our processed_query into a numerical representation called a "vector." This process is called Vectorization.

What's happening? The TfidfVectorizer (which we'll explore in Chapter 5: TF-IDF Vectorization Model) takes our cleaned query "what is machine learning" and transforms it into a list of numbers. Each number represents how important a word is in the query, based on all the documents we have. This query_vector is now ready for comparison.

3. Calculating Similarity

Now that both our query and all our documents are in numerical vector forms, we can compare them! This step finds out how "close" or "similar" the query vector is to each document vector. We use a method called Cosine Similarity.

What's happening? cosine_similarity() calculates a score for every single document in our collection, indicating how similar it is to our query_vector. A score closer to 1 means "very similar," while a score closer to 0 means "not similar at all." This gives us a list of scores, one for each document. We'll learn more about this in Chapter 6: Cosine Similarity Scoring.

4. Ranking the Results

After getting a similarity score for every document, the next logical step is to sort them! We want to show the most relevant documents first.

What's happening? ranked_indices is now a list of numbers. The first number in this list is the index of the document with the highest similarity score, the second number is the index of the document with the second-highest score, and so on. This effectively "ranks" our documents!

5. Preparing Final Results

Finally, with our ranked_indices, the Search Engine Core Logic puts together the actual information to display for each result: the document's name, a small content snippet, and its relevance score.

What's happening? This loop goes through our ranked_indices. For each index, it looks up the document's original name, its file path (which allows us to link to the actual PDF), and a brief summary of its content. It also includes the calculated similarity_score. All this information is packaged into a list of dictionaries (ranked_documents), which is then sent back to the Streamlit User Interface to be shown beautifully on the screen!

The raw document_names, document_paths, and document_contents lists are derived from our initial preprocessed_data.json file, which we'll discuss in Chapter 3: Preprocessed Document Data. The way we construct clickable paths is part of the Document Path Resolver concept.

Conclusion

The Search Engine Core Logic is the brilliant conductor of our search application. It takes your raw query, guides it through several complex but crucial steps – preprocessing, vectorization, similarity calculation, and ranking – to ultimately present you with the most relevant documents. It's the engine that makes our search truly work!

In the next chapter, we'll open the vault and look at the "raw materials" our search engine uses: the Preprocessed Document Data. This is the organized information about all the documents that the core logic can search through.

Chapter 3: Preprocessed Document Data

Chapter 3: Preprocessed Document Data

Welcome back! In Chapter 2: Search Engine Core Logic, we explored the "brain" of our search engine – the clever part that figures out how to find documents based on your query. But even the smartest brain needs good "ingredients" to work with. Imagine a master chef: they can't cook a delicious meal from raw, unprocessed ingredients that are still in their packaging! They need them peeled, chopped, and ready to go.

This is exactly what Preprocessed Document Data is for our search engine. It's the collection of all our PDF documents, but they've been carefully prepared, cleaned, and organized before anyone even types a search query.

The problem this abstraction solves is making the search process incredibly fast and efficient. Instead of having to open, read, and understand a raw PDF document every time someone searches, we do all that hard work once. We clean up the text, strip out unnecessary formatting, and store only the essential information in a neatly organized file. Think of it as a comprehensive, pre-indexed library catalog, where all the essential details about each book are already summarized and neatly organized for quick lookup.

What is Preprocessed Document Data?

Our Preprocessed Document Data is the structured information loaded from a special file called "preprocessed_data.json". This file acts like our library's digital catalog.

For each PDF document we want to search through, this catalog holds:

• Document Name: The name of the document (e.g., "Introduction to AI.pdf").
• Content: The entire cleaned text from inside the PDF, ready for searching.
• File Path: Where the original PDF file is located on your computer, so you can open it directly from the search results.

This pre-computation means the search engine doesn't have to waste time processing raw PDFs during a search. All the "ingredients" are already prepped!

How Our App Uses Preprocessed Document Data

When our application starts, one of the first things it does is load this "preprocessed_data.json" file. It's like opening the library catalog at the beginning of the day.

What's happening? The load_preprocessed_data function simply opens our preprocessed_data.json file and reads all the organized document information from it. The documents variable then holds a list of these document details. Each item in this documents list is like a record for one PDF.

After loading the data, the application needs to get specific pieces of information out of it – like all the document names in one list, all the document contents in another, and all the paths.

What's happening? We use a Python trick called a "list comprehension" to quickly pull out all the document_names into one list, all the file_paths into another, and all the contents into a third. These lists (document_names, document_paths, document_contents) are then ready to be used by the Search Engine Core Logic for processing your queries.

Under the Hood: The Journey of Preprocessed Data

Let's visualize how this "prepped" data is used:

1. Application Startup: When you run src/index.py (our Streamlit app), it kicks things off.
2. Load Data: It calls the load_preprocessed_data() function.
3. Read JSON: This function opens and reads the contents of the preprocessed_data.json file.
4. Return Structured Data: The JSON file's content is turned into a Python list of dictionaries, where each dictionary represents one document.
5. Extract Specifics: The application then goes through this list and creates separate, easy-to-use lists for just the names, just the contents, and just the file paths.
6. Ready for Core Logic: These organized lists (document_names, document_contents, document_paths) are now available globally for the Search Engine Core Logic and the TF-IDF Vectorization Model to use whenever a search query comes in.

Here's a peek at what the preprocessed_data.json file might look like internally (simplified example):

Each {...} block is a dictionary holding information for one document. This format makes it very easy for Python to read and work with.

Why is this important? Raw vs. Preprocessed

Let's look at why having this preprocessed data is such a big deal for our search engine:

The preprocessing step (which we'll explore in Chapter 4: Text Preprocessing) turns those complex raw PDFs into the simple, fast-to-use data we see here. This is a fundamental step to make our search engine perform well.

Conclusion

The Preprocessed Document Data is the well-organized "pantry" of our search engine. It ensures that all the information from our PDF documents is cleaned, structured, and immediately available for the Search Engine Core Logic to use. By doing this heavy lifting once, before any search happens, we make our search engine incredibly fast and efficient.

Now that we understand what this preprocessed data is and why it's so important, let's dive into how those raw, messy PDFs are actually turned into this neat, searchable data. In the next chapter, we'll uncover the secrets of Text Preprocessing.

Chapter 4: Text Preprocessing

Chapter 4: Text Preprocessing

Welcome back, future search engine wizard! In Chapter 3: Preprocessed Document Data, we learned that our search engine relies on "preprocessed" data – documents that have already been cleaned and organized into a special JSON file. This pre-preparation makes searching super fast!

But how do those raw, sometimes messy PDF documents get turned into that neat, tidy data? And how do we make sure that your search query, which you type into the Streamlit interface, is also as clean and ready for searching as our documents?

This is where Text Preprocessing comes in. Think of it as the meticulous "washing, peeling, and chopping" step in our cooking analogy. Just like a chef prepares ingredients to ensure consistency before cooking, our search engine prepares text to ensure that all words are in a standard, predictable form.

The problem this abstraction solves is making search accurate and consistent. Imagine if you search for "Apple" but a document says "apple!". Without preprocessing, the computer might think these are different words. Text preprocessing makes sure that variations like "Apple," "apple!", "APPLE", and "apple" are all treated as the exact same word. This makes your search results much more reliable and robust.

Let's look at a concrete example. If a user types the query "What's the best Machine Learning algorithm of 2023?", we need to clean this up so it can effectively match document content like "machine learning algorithms".

Why Clean Text Matters

When you work with text, especially from various sources like PDFs, you'll find all sorts of inconsistencies:

• Capitalization: "Apple", "apple", "APPLE"
• Punctuation: "search!", "search.", "search?"
• Numbers: "Chapter 1", "2023 report", "word123"
• Extra Spaces: " hello world "

Without cleaning, each of these variations would be seen as a completely different word by our search engine, leading to missed results. Text preprocessing fixes this by applying a set of rules to normalize the text.

The Key Steps of Text Preprocessing

Our project focuses on three main steps for text preprocessing:

1. Lowercasing: Converting all text to lowercase.
2. Removing Punctuation: Getting rid of symbols like !, ?, ., ,, etc.
3. Stripping Numbers: Removing digits like 0, 1, 2, 3, etc.

Let's see how these simple steps transform text using our example query "What's the best Machine Learning algorithm of 2023?":

After these steps, the query "What's the best Machine Learning algorithm of 2023?" becomes "whats the best machine learning algorithm of ". This cleaned version is now much more likely to match relevant content in our documents, which have undergone the same cleaning process.

How Our App Preprocesses Text

Both the documents (which are cleaned before they are loaded into preprocessed_data.json) and your user query go through this exact same cleaning process. This ensures everything is consistent.

In our src/index.py file, we have a special function called preprocess_query that handles this for the user's input:

This function is a simple but powerful tool that ensures all text fed into our search engine is standardized.

Let's test this function with our example query:

Input: "What's the best Machine Learning algorithm of 2023?"

Output: "whats the best machine learning algorithm of "

As you can see, the query is now clean and ready!

Under the Hood: Text Preprocessing in Action

When you type a query into our search engine, here's how the text preprocessing step fits into the overall process:

1. User Types Query: You enter your search request, like "Is AI becoming too smart?".
2. Streamlit Sends Query: The Streamlit User Interface sends this raw query to the Search Engine Core Logic.
3. Core Logic Calls Preprocessor: The Search Engine Core Logic knows it needs clean text, so it calls our preprocess_query function.
4. Preprocessing Steps:
   • The function takes "Is AI becoming too smart?".
   • First, it converts everything to lowercase: "is ai becoming too smart?".
   • Next, it removes the punctuation (the question mark): "is ai becoming too smart".
   • (In this example, there are no numbers, so that step doesn't change anything.)
5. Cleaned Query Returned: The preprocess_query function returns the clean text, "is ai becoming too smart", back to the Search Engine Core Logic.
6. Further Processing: The Search Engine Core Logic can now use this standardized query for the next steps, like converting it into a numerical vector (which we'll cover in the next chapter!).

The Code Behind the Cleaning

Let's break down the preprocess_query function from src/index.py step-by-step:

1. Lowercasing

This line is very straightforward! Python's .lower() method takes any string and returns a new string with all characters converted to their lowercase equivalent. For example, "Hello World" becomes "hello world". This is crucial for matching words regardless of how they are capitalized.

2. Removing Punctuation

This line uses re.sub(), which is a powerful function for finding and replacing patterns in text using regular expressions.

• r'[^\w\s]': This is the pattern we are looking for.
  • r'': Indicates a "raw string," which is good practice for regular expressions.
  • [^...]: Means "match any character that is NOT inside these brackets."
  • \w: Matches any "word" character (letters, numbers, and underscore _).
  • \s: Matches any "whitespace" character (spaces, tabs, newlines).
  • So, [^\w\s] means "match any character that is NOT a word character AND NOT a whitespace character." This effectively targets punctuation and symbols.
• '': This is what we want to replace the matched punctuation with – an empty string, effectively deleting it.

So, if query is "Hello, world!", this line would turn it into "Hello world".

3. Stripping Numbers

This is another use of re.sub():

• r'\d+': This is the pattern for numbers.
  • \d: Matches any single digit (0-9).
  • +: Means "one or more" of the preceding character.
  • So, \d+ means "match one or more digits together." This will find numbers like 1, 12, 123, 2023.
• '': Again, we replace the matched numbers with an empty string, removing them.

If query is "report 2023" after other steps, this line would turn it into "report ". If it was "chapter1", it would become "chapter".

Conclusion

Text Preprocessing is a critical foundational step for any effective search engine. It ensures that both your search queries and the content of our documents are consistently cleaned and normalized. By performing simple operations like lowercasing, removing punctuation, and stripping numbers, we make sure that our search engine can accurately match words and provide relevant results, no matter how the original text was formatted. It's like putting everyone in the same uniform so they can be easily recognized!

Now that our text is squeaky clean and standardized, the next challenge is to turn these words into something a computer can actually compare numerically. In the next chapter, we'll dive into how we convert words into meaningful numbers using the TF-IDF Vectorization Model.

Chapter 5: TF-IDF Vectorization Model

Chapter 5: TF-IDF Vectorization Model

Welcome back! In Chapter 4: Text Preprocessing, we learned how to clean up raw text, turning messy sentences like "What's the best Machine Learning algorithm of 2023?" into a clean, standardized "whats the best machine learning algorithm of ". Now that our text is squeaky clean, we have a new challenge: how do we teach a computer to understand and compare these words? Computers are excellent with numbers, but not so much with human language directly.

This is where the TF-IDF Vectorization Model comes in. Think of it as a special translator that converts our human-readable words into numerical codes, or "vectors," that a computer can easily understand and process. It doesn't just turn words into any numbers; it assigns a numerical weight to each word, indicating how important that word is in a specific document compared to the entire collection of documents.

The problem this abstraction solves is giving words a measurable "importance score" so that a computer can compare the meaning of a search query with the meaning of documents. Imagine turning all the books in a library into a set of unique barcodes, where the numbers in the barcode also reflect the specific themes or keywords of each book. TF-IDF does something similar for our words, helping our search engine find the most relevant documents.

Let's say you search for "deep learning techniques". TF-IDF will convert this into a numerical vector, where the numbers tell the computer how much "deep learning" and "techniques" contribute to the overall meaning of your query. Then, it will use the same logic for all our documents to find the best match.

What is TF-IDF? (Term Frequency-Inverse Document Frequency)

TF-IDF stands for Term Frequency-Inverse Document Frequency. It's a fancy name for a clever idea that combines two simple measurements:

1. TF (Term Frequency): How often a word appears in a single document.

   • If the word "robot" appears 10 times in Document A but only 1 time in Document B, then "robot" has a higher Term Frequency in Document A.
   • Idea: The more a word appears in a document, the more important it might be to that document's topic.

2. IDF (Inverse Document Frequency): How unique or rare a word is across all the documents in our collection.

   • Words like "the", "a", "is" appear in almost every document. They don't help much in distinguishing one document from another. They have a low Inverse Document Frequency score.
   • Words like "quantum entanglement" or "neural network" appear in only a few specific documents. These words are much more useful for identifying the topic of those documents. They have a high Inverse Document Frequency score.
   • Idea: Words that appear in fewer documents are generally more specific and thus more important for a search.

The magic happens when we combine them:

TF-IDF = TF (Term Frequency) × IDF (Inverse Document Frequency)

A word gets a high TF-IDF score if it appears frequently in a specific document (high TF) AND is relatively rare across all other documents (high IDF). This means the word is very characteristic of that particular document.

How Our App Uses TF-IDF

Our project uses TfidfVectorizer from the sklearn (scikit-learn) library to create these TF-IDF numerical vectors. It's a powerful tool that handles all the calculations for us.

There are two main steps when using TfidfVectorizer:

1. Learning from all documents (fit_transform): When our application starts, it first "learns" from all the Preprocessed Document Data. It figures out all the unique words, calculates their overall rarity (IDF scores), and then converts each document's text into a numerical TF-IDF vector. This happens once.

   What's happening?

   • vectorizer = TfidfVectorizer(): We create an instance of the TF-IDF "translator" tool.
   • tfidf_matrix = vectorizer.fit_transform(document_contents): This is the crucial step. The vectorizer first fits itself to all the document_contents. This means it reads all words, counts their occurrences, and calculates their IDF scores. Then, it transforms each document's text into a numerical vector based on these learned scores. The tfidf_matrix now holds a numerical representation for every single document. Each row in this matrix is a document's TF-IDF vector.

2. Transforming the user's query (transform): When a user types a query, we first clean it using Text Preprocessing. Then, we use the same TF-IDF vectorizer (which has already learned from our documents) to convert the cleaned query into its own TF-IDF vector. This happens every time a new query is submitted.

   What's happening?

   • query_vector = vectorizer.transform([processed_query]): We take our processed_query (e.g., "deep learning techniques") and use the already-trained vectorizer to convert it into a numerical vector. This query_vector now has numbers that represent the importance of "deep learning" and "techniques" in this specific query, relative to all the documents the vectorizer learned from. We put processed_query in [] because transform expects a list of texts, even if it's just one query.

Under the Hood: TF-IDF in Action

Let's visualize how the TF-IDF Vectorization Model fits into our search engine's workflow:

1. Application Startup: When our src/index.py application starts, it first loads all the Preprocessed Document Data, getting a list of cleaned document texts (document_contents).
2. TF-IDF Learns from Documents: The application then initializes the TfidfVectorizer and calls fit_transform() on document_contents. The vectorizer analyzes all words in all documents, computes their TF-IDF scores, and stores them in tfidf_matrix. This tfidf_matrix is essentially our library of document barcodes.
3. User Enters Query: A user types "new research papers" into the Streamlit User Interface.
4. Query Preprocessing: The Search Engine Core Logic takes this raw query and cleans it using Text Preprocessing, resulting in processed_query ("new research papers").
5. Query Transformation: The processed_query is then sent to the same TfidfVectorizer (which already learned from all documents) for transform()ation.
6. Query Vector Created: The vectorizer converts "new research papers" into a query_vector – a list of numbers representing the importance of each word in the query.
7. Ready for Comparison: Now, both our documents (in tfidf_matrix) and our query (in query_vector) are in the same numerical format, ready to be compared!

Why TF-IDF is Better Than Simple Word Counts

Imagine if we just counted how many times each word appeared.

• "The dog runs fast." (TF: the=1, dog=1, runs=1, fast=1)
• "The cat runs slow." (TF: the=1, cat=1, runs=1, slow=1)

If you search for "dog", both sentences would get a score of 1 for "runs" and "the", potentially making them seem equally relevant, even if only one is about dogs. TF-IDF helps filter out these common words:

TF-IDF helps us focus on the words that truly define the topic of a document or query, making our search more intelligent.

Conclusion

The TF-IDF Vectorization Model is the crucial step that bridges the gap between human language and computer processing. By converting cleaned text into meaningful numerical vectors, where each number reflects a word's importance, TF-IDF allows our search engine to quantitatively understand and compare documents and queries. It transforms simple words into a powerful numerical language that drives our search capabilities.

Now that both our documents and our queries are represented as comparable numerical vectors, the next logical step is to figure out how "similar" they are. In the next chapter, we'll explore Cosine Similarity Scoring, the method we use to measure this numerical resemblance.

Chapter 6: Cosine Similarity Scoring

Chapter 6: Cosine Similarity Scoring

Welcome back, future search engine expert! In our last chapter, Chapter 5: TF-IDF Vectorization Model, we accomplished something amazing: we learned how to transform both our search query and all our documents into numerical "vectors." Think of these vectors as unique numerical fingerprints, or even directions in a vast digital space, representing the meaning of the text.

Now, we have a new, crucial challenge: Once everything is turned into numbers, how do we actually compare them? How do we tell if your "deep learning" query is "similar" to a document about "neural networks" or completely unrelated to one about "ancient history"? We need a way to measure this numerical resemblance.

This is where Cosine Similarity Scoring comes in. Imagine you have two arrows pointing in different directions. Cosine Similarity is a mathematical tool that tells you how "aligned" these arrows are. The closer they point in the same direction, the more similar they are. This score is precisely how our search engine determines the relevance of each document to your query. It's like a sophisticated matching algorithm that identifies how closely aligned your search question is with the content of each document.

The problem this abstraction solves is giving us a concrete, measurable way to compare the "meaning" of your query with the "meaning" of every document. Without it, even with perfect numerical vectors, we wouldn't know which documents are the best match for your search.

What is Cosine Similarity? (The Angle of Relevance)

At its heart, Cosine Similarity calculates the cosine of the angle between two vectors. Don't worry if that sounds complicated; let's break it down with a simple analogy:

Imagine each document and your search query as an arrow (a vector) starting from the same point, pointing into a huge, multi-dimensional space.

• If two arrows point in almost the exact same direction: The angle between them is very small (close to 0 degrees). This means they are very similar in meaning. The Cosine Similarity score will be close to 1 (its highest possible value).
• If two arrows point in very different directions, almost perpendicular to each other: The angle between them is around 90 degrees. This means they are not very similar. The Cosine Similarity score will be close to 0.
• (If they point in opposite directions, it would be -1, but with TF-IDF vectors, scores are typically between 0 and 1.)

Why is this helpful? This method focuses purely on the direction of the vectors, not their length. Why is this important? Because a longer document might have higher TF-IDF values just because it has more words, not necessarily because it's more relevant. Cosine Similarity ignores this "length" factor and focuses only on whether the themes (directions) represented by the words are aligned.

How Our App Uses Cosine Similarity

After the TF-IDF Vectorization Model has done its job, we have two key pieces of data ready for comparison:

1. Your search query, transformed into a numerical query_vector.
2. All our preprocessed documents, collectively represented as a tfidf_matrix (where each row is a document's vector).

Our project uses a function called cosine_similarity from the sklearn.metrics.pairwise library to perform this calculation efficiently.

Let's look at the specific line of code from src/index.py that makes this magic happen:

What's happening?

• cosine_similarity(query_vector, tfidf_matrix): This function takes your single query_vector and compares it to every single document vector inside the tfidf_matrix. For each comparison, it calculates a Cosine Similarity score.
• .flatten(): The cosine_similarity function normally returns its results in a slightly complex format (a 2D array, even if one dimension is just 1). .flatten() simply converts this into a simple, single list of numbers. Each number in this similarity_scores list corresponds to one document's relevance score.

Example Input/Output:

If your query_vector represents "artificial intelligence" and tfidf_matrix contains vectors for three documents (Doc A, Doc B, Doc C):

• query_vector: (numbers representing "artificial intelligence")
• tfidf_matrix:
  • Row 1 (Doc A): (numbers representing "machine learning concepts")
  • Row 2 (Doc B): (numbers representing "ancient Roman history")
  • Row 3 (Doc C): (numbers representing "AI ethics and future")

The similarity_scores would look something like this: [0.85, 0.02, 0.91]

• 0.85: High similarity with Doc A ("machine learning concepts").
• 0.02: Very low similarity with Doc B ("ancient Roman history").
• 0.91: Very high similarity with Doc C ("AI ethics and future").

These scores directly tell us how relevant each document is to your query!

Under the Hood: The Relevance Matchmaker

Let's trace how Cosine Similarity fits into the overall search process when you enter a query:

1. User Enters Query: You type your search, say "future of AI."
2. Query to Core Logic: The Streamlit User Interface sends this to the Search Engine Core Logic.
3. Preprocessing: The core logic cleans your query using Text Preprocessing.
4. Vectorization: It then sends the cleaned query to the TF-IDF Vectorization Model to get its numerical query_vector.
5. Document Vectors Ready: The core logic already has the tfidf_matrix (all document vectors) from when the app started.
6. Similarity Calculation: Now, the core logic sends both the query_vector and the tfidf_matrix to the Cosine Similarity Calculator (our cosine_similarity function).
7. Scores Returned: The calculator computes a similarity score for each document and returns the similarity_scores list.
8. Ranking: The core logic uses these scores to sort (rank) the documents from most relevant to least relevant.
9. Results Display: Finally, the ranked document information is sent back to the Streamlit User Interface to be displayed to you!

Deeper Dive: The cosine_similarity Function

The beauty of using sklearn is that the complex mathematics of calculating the cosine of angles between high-dimensional vectors is handled for us. You don't need to manually implement the formula, which involves dot products and vector magnitudes.

The cosine_similarity function efficiently performs this pairwise comparison. It's designed to be fast and accurate, even with many documents and complex vectors.

The core idea for two vectors, A and B, is:

Cosine Similarity (A, B) = (A ⋅ B) / (||A|| ⋅ ||B||)

• A ⋅ B: This is the "dot product" of the vectors, a way to measure how much they "overlap."
• ||A|| and ||B||: These are the "magnitudes" (lengths) of the vectors.

By dividing the dot product by the product of their magnitudes, we normalize the score, so it's only about the angle and not the length. This ensures our relevance scores are fair, regardless of how long or short a document is.

Conclusion

Cosine Similarity Scoring is the "relevance meter" of our search engine. By calculating the cosine of the angle between your query's numerical vector and each document's numerical vector, it provides a precise, normalized score that tells us how closely related they are. This critical step allows our search engine to move beyond simply finding keywords and instead understand the conceptual similarity between your question and our documents, giving you truly relevant results.

Now that we know how to rank documents based on relevance, the final piece of the puzzle is making sure we can actually access those documents! In the next and final chapter, we'll explore the Document Path Resolver, which helps us create clickable links to open the original PDF files.

Chapter 7: Document Path Resolver

Chapter 7: Document Path Resolver

Welcome to the final chapter of our project tutorial! In our previous chapter, Chapter 6: Cosine Similarity Scoring, we mastered how to rank documents based on their relevance to your search query. We now have a list of the most important documents, along with their names and a snippet of their content. That's fantastic!

But there's one crucial step missing: How do you actually open the PDF file you found? When you see a search result, you expect to click on its title and have the document magically open on your computer. If our search engine just gave you a simple path like "data/documents/report.pdf", your browser wouldn't know what to do with it.

This is exactly the problem the Document Path Resolver solves. Think of it as a helpful postal service for your digital documents. It takes a simple, relative address (like "turn left at the bakery") and turns it into a precise, full, and universally understood address (like "123 Main Street, Cityville, State, Zip Code") that your computer's browser or PDF viewer can use to find and open the correct file, every single time. It ensures that when you click on a search result, the corresponding PDF document opens without any issues.

Let's imagine you search for "quantum physics". Our search engine finds a relevant document called "Quantum_Basics.pdf" stored in a folder called data/documents. The Document Path Resolver's job is to take this internal path and convert it into a clickable URL that looks something like file:///C:/Users/YourName/IR_CW/data/documents/Quantum_Basics.pdf.

Why Do We Need a Path Resolver? (The Challenge of Local Files)

When you browse the internet, links usually start with http:// or https://. These tell your browser to go find a file on a remote server. But our project deals with PDF files stored right on your local computer.

For a web browser to open a file from your own computer, it needs a special kind of link called a file:// URL. This URL tells the browser, "Hey, don't look on the internet; look right here on this computer."

Creating these file:// URLs isn't as simple as just sticking file:// in front of the file path. We face a few challenges:

1. Relative vs. Absolute Paths: Our preprocessed_data.json might store paths like data/documents/Report.pdf. This is a relative path (relative to where our project folder is). We need to convert it into an absolute path that starts from the very root of your hard drive (e.g., C:\ on Windows or / on Linux/macOS).
2. Operating System Differences: Windows uses backslashes (\) in paths (e.g., C:\folder\file.pdf), while Linux and macOS use forward slashes (/) (e.g., /home/user/folder/file.pdf). file:// URLs prefer forward slashes for consistency.
3. Special Characters in File Names: What if a PDF file is named "My Report (2023).pdf"? The spaces and parentheses are special characters in URLs and need to be "URL-encoded" so the browser understands them correctly.

The Document Path Resolver handles all these details for us!

How Our App Uses the Document Path Resolver

When our application starts, it loads all the Preprocessed Document Data. As part of this loading process, for every document, it immediately resolves its file_path into a complete, clickable URL.

Let's look at the relevant code from src/index.py:

What's happening? This block of code iterates through each document record. For each document, it takes its file_path (which is stored as a relative path in preprocessed_data.json) and performs a series of transformations. It joins it with a base_directory, cleans it up for the operating system, ensures forward slashes, encodes special characters, and finally adds the file:/// prefix. The resulting final_clickable_url is then stored in our document_paths list, ready to be used as the href attribute in the HTML link on our Streamlit User Interface.

Under the Hood: Resolving a Document Path

Let's visualize the journey of a document's relative path to become a clickable URL:

1. Application Startup: When src/index.py starts, it reads preprocessed_data.json and gets a relative_path for each document.
2. Path Resolution: For each relative_path, the Document Path Resolver component (the code block we just saw) performs all the necessary steps to convert it into a final_clickable_url.
3. URL Stored: This final_clickable_url is stored in the document_paths list, ready for use.
4. Display Link: When search results are displayed by the Streamlit User Interface, it uses this final_clickable_url to create a working <a href="..."> link.
5. User Clicks: When you click the link in the search results, your browser or operating system uses this perfectly formatted file:// URL to locate and open the PDF.

Deeper Dive into the Code Steps

Let's break down each line of the path resolution process:

1. Setting the base_directory

The base_directory is the crucial starting point. It's the absolute path to the folder where your data folder (containing pdfs) is located. The r before the string means it's a "raw string," which is good for Windows paths to avoid issues with backslashes. You must ensure this base_directory points to the correct location on your computer for the links to work!

2. Joining Paths with os.path.join()

The os.path.join() function is smart. It takes different parts of a path and intelligently combines them, using the correct slash (\ or /) for your operating system. It handles cases like whether base_directory ends with a slash or not.

3. Normalizing Paths with os.path.normpath()

os.path.normpath() cleans up the path. It resolves any . (current directory) or .. (parent directory) components and removes redundant slashes. It makes sure the path is in its simplest, most consistent form for your operating system.

4. Replacing Backslashes for URLs

URLs (especially file:// URLs) generally prefer forward slashes (/), even on Windows. This simple replace() method ensures that our path uses the correct slash direction for a web context.

5. URL-Encoding with urllib.parse.quote()

This is a very important step! urllib.parse.quote() takes a string and replaces any "unsafe" characters (like spaces, parentheses, question marks, etc.) with their %hex equivalent. For example, a space becomes %20. This ensures that when your browser sees the URL, it correctly interprets the filename, even if it contains unusual characters.

6. Adding the file:/// Prefix

Finally, we add the file:/// scheme at the beginning. This tells the browser that it's dealing with a local file path, not an internet address. Notice the three slashes: file:// is the scheme, and the third slash is part of the absolute path, typically representing the root of the file system.

Why All This Effort? (Raw vs. Resolved Paths)

Let's compare a raw path to a resolved path:

The Document Path Resolver is a small but critical component that transforms internal data into user-facing functionality, making our search engine not just smart, but also practical and user-friendly!

Conclusion

The Document Path Resolver is the unsung hero that ensures our powerful search engine delivers a complete experience. It takes the simple relative paths of our documents and transforms them into perfectly formatted, universally understood file:// URLs. By handling the complexities of absolute paths, operating system differences, and URL encoding, it guarantees that when you click on a search result, the correct PDF document opens seamlessly in your browser or viewer. It’s the final bridge connecting our digital search results to the physical files on your computer.

With this chapter, you now have a comprehensive understanding of all the key components that make up our PDF Search Engine, from the user interface to the core logic that retrieves and opens your desired documents. Congratulations on completing the tutorial!