In this lesson, we'll discuss
using HTML scraping and cached pages
as techniques for gathering information during a pentest.
HTML scraping involves collecting data
from website's HTML code.
When you right-click on that webpage and select "inspect,"
you'll see the underlying code
that guides the web browser in presenting the information.
This process can uncover hidden
or overlooked details within a site's markup.
For example, comments within HTML code
or meta tags might reveal backend technologies,
server types, or even internal names
and IP addresses that are not meant to be public.
To start HTML scraping,
you can use tools like Beautiful Soup and Python
or directly examine the source code in the browser.
Pentesters can look for comments
or script text left by developers that could reveal paths
to restricted directories, backend software versions,
or configuration details useful for targeted attacks.
Imagine you're gathering information
on Acme Corp's website application at www.acmecorp.com.
By inspecting the HTML source code,
you might find comments like:
"<!-- Note: Update server to version 2.0 next month -->"
or <meta name="generator"
content="Drupal 8.9">"
These details can hint at potential vulnerabilities
such as known exploits for the indicated Drupal version.
Drupal is a content management software
used for managing websites
and this particular version was vulnerable
to a remote code execution exploit
that was not patched until version 8.9.10.
Next, let's touch on how to use cached pages
to gain valuable information.
When information has been removed from a website,
cached pages can be invaluable.
Search engines like Google store cached pages of websites,
and services like Wayback Machine
archive webpages periodically.
These resources can help you access content
that has been deleted or altered.
For example, to use the cached feature on Google
to find the old wiki page of Albert Einstein,
you can simply type in: 'cache:
and then the full URL for that Wiki page
of Albert Einstein, as shown here.
At the top, you'll see that Google identifies the timestamp
when the page was cached.
On cached or archive pages,
you can view the HTML code
for comments and meta information
that might not be visible or rendered on the page.
We could also use HTTP header sniffing tools
which can provide valuable insights.
For instance, just say we're looking for a website
that was moved called Kelly Innovations.
We might get a response like this.
This response indicates a redirection
and reveals a server type and version
which is Apache/2.4.29 on Ubuntu.
Again, helping us to key in
on what type of vulnerability to look for,
for what type of software.
So remember, HTML scraping and exploring cached pages
are powerful techniques for pentesters.
They help uncover valuable information
that might not be immediately visible
or accessible due to web updates or deliberate removal.
By utilizing these methods,
pentesters can gain a comprehensive understanding
of their target's web presence
and potential vulnerabilities.