HTTP / Web Recon With Curl, Wget, Robots, & .git checks
Quickly probe web app endpoints, retrieve robots/sitemap, test parameters.
curl -s "http://<target>/robots.txt" -o - | sed -n '1,200p'
Pull robots.txt, print first 200 lines - quick mapping of disallowed paths.
curl -s "http://<target>/sitemap.xml" -o - | sed -n '1,200p'
Grab sitemap to enumerate public URLs.
curl -i "http://<target>/login.php?username=username&password=password"
Manual GET request with query parameters to observe response/redirects (useful for testing injection point and behaviour). Remember to change “username” and “password” inside the string after both equal symbol with the correct username and password located on the target system.
curl -s "http://<target>/.git/HEAD" -o - | sed -n '1,200p'
Check for exposed .git repository HEAD (common leakage).
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://<target>/
Mirror a web site locally for analysis. Remember to use ethically.