Tom Evslin posted about his experiences working with Gerard Salton in the early 1960's.

Everybody assumed that the best results would be obtained by algorithms which made an attempt at understanding English syntax. (which is very hard to do). WRONG! Turns out that syntax was a waste of time; all that matters is semantics – the actual words used in the query and the documents – not how they relate to each other in a sentence. Sometimes it was (and still is) useful to search for phrases as if they were words. But you get that just by observing word order or how close words are to each other – not trying to parse sentences.

Modern search engines may use quite a large amount of user tracking and heavily emphasize linkage data, but if you want to see the roots of search I highly recommend reading Salton's A Theory of Indexing.