A simple C# web crawler
To use
ICrawlWebSites crawler = new WebCrawlerService();
SiteMap sitemap = crawler. CrawlWebSite(“SomeUrl”);
Test the crawler against a real web-site.
Test behaviour using stubbed web-sites. Obviously needs a lot more tests.