Latent Semantic Indexing and Information Retrieval

Most common search engines have serious problems returning all the documents which are important to a given user query because they can not disambiguate ambiguous terms or find documents which only include synonyms of the query terms. A promising approach to over-come these shortcomings gives Latent Semantic Indexing (LSI). This indexing scheme uses Singular Value Decomposition (SVD) to reveal the underlying latent semantic structure of documents. The implementation described in this book is a local search engine called Bosse for Wikipedia articles. Four different search types were implemented which allow to search for documents or terms similar to a given term, query or document. These search types are evaluated and the importance of term weighting, exclusion of non content words and the optimal number of remaining dimension (k) during SVD are discussed. Furthermore, an introduction to Latent Semantic Indexing (LSI) and an explanation of the Singular Value Decomposition (SVD) is given.