Lecture Notes on the Deep Web and Dark Web

Introduction to the Deep Web

• Definition: The Deep Web comprises parts of the web that Google cannot access.
• Examples of Deep Web Content:
  • Password-protected pages (e.g., bank information, private social media profiles).
  • Dynamic content generated by user interactions (e.g., personalized search results on Amazon, Facebook feeds).
  • Pages excluded by the robots.txt file, which instructs search engines not to index certain pages.
  • Content never linked or discovered by search engines.

Key Characteristics of the Deep Web

1. Password-Protected Content:

   • Google cannot access pages requiring passwords.
   • E.g., private Facebook profiles versus public pages.

2. Dynamic Content:

   • Content changes based on user input or behavior.
   • Search results on retail sites like Amazon are not fully indexed by Google.

3. Robots.txt Standard:

   • Websites use this file to request search engines not to index specific pages.
   • Trust-based system; potential to bypass it but may violate norms.

4. Unlinked Content:

   • Pages without external links remain undiscovered by search engines.
   • E.g., personal web servers accessible only via IP address.

Size of the Deep Web

• Google's index covers approximately 5% of the web (around 1 trillion pages).
• The Deep Web is estimated to be 95% of the total web content, significantly larger than the surface web.
• Many resources such as academic literature and forums are behind passwords.

Transition to the Dark Web

• Definition of the Dark Web:
  • A subset of the Deep Web, constructed differently for anonymity.
  • Built using different protocols and languages; e.g., the Tor network (onion routing).

Understanding the Dark Web

• Tor Network:

  • Allows access to .onion sites, which are not indexed by traditional search engines.
  • Uses layered encryption (like an onion) to protect user privacy.
  • Routes requests through multiple servers to mask user identity.

• Anonymity and Privacy:

  • Facilitates both good and bad activities.
  • Examples include:
    • Positive: Whistleblowing, communication in oppressive regimes.
    • Negative: Illegal activities like drug trafficking (e.g., Silk Road).

Security and Risks

• Using Tor Safely:
  • Avoid using certain technologies (e.g., Flash) that can expose user location.
  • The nature of anonymity can lead to potential exploitation but is not inherently dangerous.

Differences Between the Web, Deep Web, and Dark Web

• Web: Standard internet accessible by everyone.
• Deep Web: Normal content behind passwords or unlinked pages; not inherently dangerous.
• Dark Web: Specialized area of the Deep Web, emphasizing anonymity and privacy through unique structures.

Conclusion

• Important to distinguish between the surface web, Deep Web, and dark web.
• Recognize the potential for both positive uses of anonymity and risks associated with the dark web.