DigitalDomainKit's Sitemap Generator creates a draft sitemap.xml from a public web page that you provide. Enter a public http:// or https:// starting URL, choose a maximum page limit, set the visible sitemap tag options, and the tool requests pages from the DigitalDomainKit server while the browser manages the crawl queue. Successful same-host pages are turned into a copyable XML sitemap preview.
This is a practical sitemap drafting tool, not an unlimited SEO crawler and not a guarantee that every indexable URL on a website has been found. The current implementation can discover ordinary links in fetched HTML, normalize URLs, remove common tracking parameters, and limit crawling to the same hostname. It does not log in, execute full browser rendering for every page, submit forms, read private dashboards, follow server redirects automatically, or prove that the generated XML is the complete canonical sitemap for a large site.
What the Sitemap Generator Actually Does
The reviewed implementation combines browser-side queue management with a Laravel Livewire server method named crawlPage. The browser starts with the URL you enter, normalizes it with the JavaScript URL API, strips hash fragments, removes common tracking parameters such as utm_source, gclid, and fbclid, and keeps only URLs on the same hostname as the starting page. Each queued URL is then passed to the server-side crawler one at a time.
On the server, the crawler validates the submitted URL with the shared public-target validator before making the HTTP request. That validator accepts public HTTP and HTTPS URLs, rejects malformed input, blocks localhost-style names, checks DNS records, and refuses private, local, link-local, and reserved IP targets. The server request uses a custom user agent, a 10-second timeout, SSL verification disabled, and automatic redirect following disabled. That last detail matters: if a URL returns a redirect status instead of a successful response, the sitemap generator records a failed or non-200 result for that URL rather than automatically chasing the destination.
When a page returns a successful response, the server extracts the HTML title and the raw anchor href values. The browser resolves relative links against the current page URL, keeps HTTP and HTTPS links on the original hostname, skips URLs it has already visited, and adds new internal URLs to the queue until the queue empties, the selected page cap is reached, or you stop the crawl. The final XML output includes only crawled pages with status 200.
How to Use the Tool
- Open the Sitemap Generator and find the Website Base URL field in the configuration panel.
- Enter a public website URL. A complete URL such as
https://example.comis safest, although the browser-side starter also prependshttps://when you enter a bare domain. - Choose the Max Pages Limit: 50, 100, 200, or 500 pages. The default value in the inspected page is 100.
- Review the sitemap tag controls. The visible options include Include lastmod tag, Include priority tag, Auto Priority (Path Depth), and a Change Frequency selector.
- Select Generate Sitemap. While the crawl runs, watch the processed-page count, queue count, discovered-link count, loaded-data estimate, current URL, progress bar, and crawl log.
- Use Stop & Build if you want to end the crawl early and build XML from the successful pages already crawled.
- After completion, inspect the Sitemap XML File, Discovered Links Table, and Technical Overview tabs before copying or downloading
sitemap.xml.
Inputs, Outputs, and Limits
The tool is most useful for small to medium public sites where important pages are reachable through ordinary HTML links. It can be a helpful first pass before you edit a production sitemap, but the output should still be reviewed against your CMS routes, canonical URLs, robots rules, server redirects, and search-console data.
| Area | Verified behavior | Important limitation |
|---|---|---|
| Starting input | One public HTTP or HTTPS website URL entered in the Website Base URL field. | Private networks, localhost, reserved IP ranges, invalid hosts, unsupported schemes, and unresolved hosts are rejected by public-target validation. |
| Page limit | The visible Max Pages Limit selector offers 50, 100, 200, and 500 pages. | The cap stops the crawl even if more internal URLs are discoverable. |
| Crawl scope | The browser queues resolved HTTP and HTTPS links only when the hostname matches the starting hostname. | Subdomains, alternate domains, CDN hosts, external stores, and separate localized domains are outside the same-host crawl unless they are the starting host. |
| Redirects | The server fetch disables automatic redirect following. | A redirecting URL may not become a sitemap entry because only status 200 pages are written to the XML output. |
| Discovery method | The server parses fetched HTML anchor tags and returns their href values to the browser queue. |
Links loaded only after complex JavaScript execution, form submissions, authenticated navigation, or API calls may be missed. |
| XML output | The generated file uses the standard sitemap urlset namespace and writes a loc entry for successful pages. |
The tool does not prove that every URL is canonical, indexable, allowed by robots.txt, or suitable for publication. |
| Request timeout | Each server fetch uses a 10-second timeout in the reviewed implementation. | Slow pages, rate limits, blocking rules, DNS problems, or unreachable servers can produce skipped entries. |
Generated Sitemap Fields
The XML builder starts with an XML declaration and a <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> element. For each crawled page whose status is 200, it writes a <url> block with the escaped URL in <loc>. If last modified output is enabled, it writes a <lastmod> value using the date returned by the server crawl, which is the crawl date in the inspected code rather than a value read from the target page's server headers.
The visible priority controls deserve a careful read. With Auto Priority (Path Depth) enabled, the current browser logic assigns 1.0 to the root path, 0.8 to one path segment, 0.6 to two path segments, and 0.4 to deeper paths. With that control disabled, pages use 0.5. These values are estimates based on URL depth, not search-engine importance scores and not recommendations from your analytics data.
The Change Frequency selector includes always, hourly, daily, weekly, monthly, yearly, and never, and the table displays the selected value for each crawled row. The XML code contains a branch for changefreq, but the inspected live JavaScript did not expose a matching initialized includeChangefreq property in the visible state. Review the generated XML preview rather than assuming the table value was written into the downloaded file.
Example Crawl Result
A conservative test case is https://example.com. The page is public, small, and stable enough to show the generator's basic flow without risking a heavy crawl. In the current logic, the submitted URL is normalized, sent to crawlPage, and accepted as a public HTTPS target. If the page returns a successful response, the crawler records a title, status 200, response size, crawl-date lastmod, and any anchor links found in the HTML. Because example.com is intentionally simple, the generated sitemap may contain only the starting URL if no same-host crawlable links are added to the queue.
The corresponding XML shape is straightforward: one <url> block for each successful crawled URL, escaped inside <loc>, optionally with <lastmod> and <priority> depending on the visible settings and the actual preview. Use this example as a baseline for understanding output structure, not as a benchmark for larger sites. Real websites often have redirects, canonical variants, pagination, nofollow links, blocked sections, JavaScript navigation, and pages that should not be published in a sitemap even if a crawler can discover them.
Public-Target Validation and Responsible Use
The Sitemap Generator makes server-side HTTP requests from DigitalDomainKit, so it includes public-target validation before the server fetch. The implementation imports App\Support\PublicNetworkTarget and calls PublicNetworkTarget::httpUrl($url) inside crawlPage. A previously verified invalid-input check for this repair showed crawlPage('http://127.0.0.1') returning success=false, which is the expected behavior for a loopback address. The live page itself returned 200 OK during review on September 16, 2026.
Use the tool for public websites you own, administer, or have permission to audit. Do not use it for intranet hosts, router interfaces, cloud metadata addresses, staging systems that resolve privately, signed URLs, customer portals, session-bearing links, unpublished previews, or credentials in query strings. The browser interface can display submitted URLs, discovered URLs, page titles, and crawl logs, and those details may remain in your browser history, clipboard, screenshots, or page state while you work.
Why the Result May Differ From Your Production Sitemap
A production sitemap is usually built from your source of truth: CMS content records, route definitions, ecommerce product data, locale configuration, or a deliberate list of canonical URLs. This generator works from reachable public links instead. That is useful for quick discovery and auditing, but it means the result can be incomplete or include URLs that you would not intentionally submit to search engines.
| Difference | Why it happens | What to check before publishing |
|---|---|---|
| Missing important page | The page may not be linked from crawled HTML, may require JavaScript, or may appear after the selected page limit. | Compare with CMS exports, route lists, navigation menus, analytics landing pages, and Search Console data. |
| Redirect URL skipped | The server fetch does not automatically follow redirects, and XML output includes only status 200 pages. | Test redirecting URLs with the Redirect Checker, then crawl the final preferred URL. |
| Duplicate-looking URLs | Query strings, case differences, trailing-slash rules, and canonical variants can create separate normalized URLs. | Review canonical tags, server redirects, URL parameters, and CMS permalink settings. |
| Unwanted page discovered | Any same-host HTML link can be queued if it passes the browser-side filters. | Remove login pages, filtered search URLs, account pages, cart URLs, and thin utility routes before publication. |
| Lastmod differs from your CMS | The current server result uses the crawl date rather than reading each page's real content update timestamp. | Replace lastmod with authoritative modified dates from your CMS, deployment records, or database when available. |
Troubleshooting
| Problem | Likely cause | What to do next |
|---|---|---|
| The crawl skips the first URL | The URL may be malformed, private, local, unresolved, blocked, timed out, or returning a non-success status. | Enter a full public https:// URL, test the same URL in a status checker, and avoid private addresses. |
| The generated XML has fewer URLs than expected | The crawl may have hit the page limit, skipped redirects, missed JavaScript-only links, or stayed within one hostname. | Increase the page limit carefully, crawl final canonical URLs, and compare against your CMS or server route list. |
| The progress bar stops before the whole site is covered | The queue may be empty, the selected limit may be reached, or failed pages may not expose additional links. | Start from a higher-level public page and make sure important sections are linked from crawlable HTML. |
| Some rows show failure or non-200 status | Those URLs did not return a successful response to the server-side request. | Inspect the URL directly, check redirects, firewall rules, rate limits, and whether the route is actually published. |
| The XML preview lacks expected optional fields | Visible table data and XML inclusion logic are separate, and some options depend on the current JavaScript state. | Always inspect the XML preview before copying or downloading, then edit the file if needed. |
Related Tools
Use the Redirect Checker before sitemap cleanup when a discovered URL returns a 301, 302, 307, or 308 and you need to find the final public destination. Use the HTTP Status Code Checker for a quick one-URL response check without running a crawl. Use the Sitemap Visualizer after you already have a sitemap URL or XML file and want to inspect structure. Use the Robots.txt Generator when you need a draft robots file that points search crawlers toward your sitemap. Use the Canonical URL Generator when you are preparing canonical tags for preferred URLs before adding them to a sitemap. Use the Keyword Cloud Density Checker after URL discovery when you want a quick look at repeated visible terms on a public page.
Technical References
- Sitemaps XML protocol
- Google Search Central sitemap overview
- MDN documentation for the HTML anchor element
- MDN documentation for the URL interface
- HTTP Semantics specification
Editorial and Technical Notes
Sitemap Generator is built and maintained by Digital Domain Kit. This draft was last reviewed on September 16, 2026 against the live page at /tool/sitemap-generator, the tool registry metadata, the rendered JavaScript behavior, the Livewire server component, and the shared public-target validator. The live page returned 200 OK during review, and the server-side crawl implementation was verified to block local targets such as http://127.0.0.1.
The verified technical method is a browser-managed same-host crawl queue backed by a Livewire server fetch for each candidate URL. The server validates public HTTP(S) targets, sends a GET request with automatic redirect following disabled, parses HTML titles and anchor links, and returns page metadata to the browser. The browser builds a table, technical overview, and XML sitemap preview from crawled rows. To report a mismatch between this documentation and the live tool, use the Digital Domain Kit contact page and include the public URL tested, selected page limit, time of the test, and any visible crawl log message.