4

you’d want to create a search index for this

this is how much ai “search tools” and how google search works

basically a crawler visits webpages and extracts useful keywords and then put them into an index

this way the search content is much shorter - like 1% of the original size. and the search happens very quickly and cheaply.

depending on the size of the data to be searched through and its complexity you may also need to bucket it and do the search in a few steps.

  1. find the relevant bucket
  2. search the items within the bucket

and potentially a few layers of buckets.

** you’d only need to do this bucketing process if you had more data than the ai was limited to

alternatively just use a search tool that handles this whole process for you like algolia…

for a pdf you’d need to use an ocr to extract the content first, then index it