Domain Fetching Config

When you add a domain to Domain Locker, the app automatically fetches information about it from WHOIS servers and other sources where available. This includes details like registration dates, expiration dates, nameservers, registrar info, and more.

You can understand how this works from a technical perspective in the How Add Domain Works dev docs.

At a high-level, it calls an API endpoint which looks up info from various sources, including registrar, whois, SSL, DNS, subdomains and more. And then auto-fills this into the form, for you to review and save.

Important: Before you can save a domain, it must have a registrar and expiration date set. If this cannot be fetched automatically, you'll need to enter it manually.

Note that:

There are some features which won't work out-of-the-box in the self-hosted instance without additional configuration (since they rely on third-party services)
For some domains and TLD types, certain data may not be publicly available and will need to be entered manually.

Keeping Domain Data Updated

Domains can be updated to fetch the latest info by sending a POST request to the /api/domain-updater endpoint. To keep domains-up-to-date automatically, set up a cron job to call this endpoint periodically.

For example:

( crontab -l 2>/dev/null; \
  echo "0 3 * * * curl -s -X POST http://app:3000/api/domain-updater"; \
  echo "0 4 * * * curl -s -X POST http://app:3000/api/expiration-reminders" ) | crontab -

You can see more about how to do this, as well as about setting up change and expiration notificiations here.

Fetching Subdomains

Subdomain discovery is powered by external services. Domain Locker supports two options, either Shodan or DNSDumpster.

Both services work by scanning the internet and building databases of domains, subdomains, IP addresses, and open ports. Domain Locker queries these services to find subdomains associated with your domain.

Setting Up Subdomain Fetching

To enable subdomain fetching, you need an API key from at least one of these services.

Option 1: Using DNSDumpster (Recommended for Self-Hosting)

DNSDumpster is free and works well for most use cases.

DNS_DUMPSTER_TOKEN=your-api-key-here

Option 2: Using Shodan

Shodan provides more detailed data but requires a paid subscription for API access.

SHODAN_TOKEN=your-api-key-here

Using Both Services

By default, Domain Locker will use whichever service is configured, or both if available (for better coverage).

If both services are configured, you can choose which one is used, by setting the DL_PREFERRED_SUBDOMAIN_PROVIDER environment variable, to either:

dnsdump - Only use DNSDumpster
shod - Only use Shodan
both - Query both services and merge the results (most comprehensive)

Unsupported Domains

Not all domains can have their data fetched automatically. Here are some common cases where fetching might fail:

Restricted TLDs
- Some ccTLDs (like .cn, .ru) restrict WHOIS access. Same with certain newer TLDs, like .horse
Domains with privacy protection
- Most registrats now implement a privacy services, so some owner details may be hidden
Recently registered domains
- WHOIS data may take time to propagate for new domains
Domains with flakey WHOIS servers
- Some domains have really unreliable WHOIS servers, that can be slow, time out, or limit requests from certain IPs.

Unfortunately there's not much that can be done by us in these cases, and the only option is to enter the data manually.

Resolving Data Fetching Issues

Check the logs

See the Checking logs docs for instructions on how to view your app logs. Within the logs you'll likely see a message indicating what went wrong, and why.

Check Your Environment Variables

See the Docker docs for how to set environment variables.

You can either set these in your .env file, in the docker-compose.yml, directly in your CLI environment, or use a secrets manager of your choice. You can verify which env vars are set correctly, by visiting the /advanced/debug-info page. For docs on how environment variables work, see Environmental Variables.

Check Service Availability

If you're using any of the third-party or managed services, then check the Service Status page

Further Debugging

Check the Debugging Guide for further steps on how to troubleshoot issues.

Resolving Subdomain Issues

You can test the subdomain endpoint, with:

curl "http://localhost:5173/api/domain-subs?domain=bbc.com"

You can check your keys are valid, with:

# Check Shodan API key is valid
curl "https://api.shodan.io/dns/domain/example.com?key=${SHODAN_TOKEN}"

# Check DNSDumpster API key is valid
curl -H "X-Api-Key: ${DNS_DUMPSTER_TOKEN}" "https://api.dnsdumpster.com/domain/example.com"