Indexing PDFs, URLs & Code (Pro)

Wyregraf Pro adds three content sources beyond your site's own posts: an uploaded PDF, any public URL, and a zip of code files. All three live in the same embedding editor from Creating Embeddings — choose the source in the Content Source dropdown on the Index tab, and that source's own fields appear below it.

Whichever source you use, give the Metadata tab a moment of care: a scraped page or an uploaded file rarely carries a good reader-facing title of its own, and the Metadata fields are what a chat shows when it cites this content.

PDF #

1. Set Content Source to PDF – Smalot.
2. Upload PDF picks the file from your media library (the uploader only accepts PDFs). Wyregraf extracts the text and reads the document's title and page count for you — the filename fills in when the PDF has no title of its own.

Two kinds of PDF won't index: scanned/image-only PDFs, which contain no extractable text, and password-protected PDFs, which can't be opened — remove the password and re-upload.

When a PDF is cited in chat, a same-origin file opens in Wyregraf's built-in viewer with page
navigation, thumbnails, and fullscreen controls. Browsers do not allow that viewer to fetch every
cross-origin file safely, so a PDF hosted on another origin opens in a new tab instead. The citation
still works; only the inline viewing experience changes.

URL Scrape #

1. URL — the web page to index. Each embedding scrapes that one page; it doesn't crawl the rest of the site, so a section of several pages is several embeddings.
2. Selector (optional) — limit the scrape to part of the page: an HTML tag (body is the default; main or article are common), or a CSS selector like #content or .entry.
3. Remove Selectors (optional) — a comma-separated list of CSS selectors to strip before indexing. Use it to drop navigation, sidebars, and cookie banners so they don't pollute the index.

The URL must be a public web address — for everyone's safety Wyregraf refuses to fetch private or internal addresses (localhost, LAN IPs, non-standard ports), and it re-checks every redirect. If your URL is rejected, see Why is my URL "not allowed"?. If the scrape reports no content found instead, the selector didn't match anything on the page — try the default body first, then narrow.

Code #

1. Set Content Source to Code, then Upload Zip with your project's zip file.
2. The exclusion fields ship prefilled so the obvious noise never reaches the index: Exclude Directories (.git, node_modules, vendor, tests, …), Exclude Files (lockfiles and dotfiles), and Exclude Filetypes — a long list of binary and asset extensions. Note that js and json are on the default filetype exclusion list; remove them from the list if your JavaScript should be indexed. Exclude Path and File takes exact paths (like plugin/src/index.php) for one-off exclusions.
3. Max File Size skips files larger than the limit (default 112000 bytes, about 112 KB) and Minimum File Size (default 200 bytes) skips near-empty stubs. Minified .min.js / .min.css files are always skipped.

Keeping them current #

Unlike post content, these sources don't live in WordPress — when the PDF, page, or codebase changes, upload or point at the new version and hit Reindex on the Index tab. Summaries & Questions work on these embeddings exactly as they do on post content, and help most on long documents.