Flatscraper - A Simple Web Scraper for Flat Listings 🔍🏠

Flatscraper is a lightweight web scraper that extracts flat listings from a specified URL. It leverages the BeautifulSoup library to parse HTML and capture essential details about flats—title, price, and location.

🚀 Features

• 🏠 Scrapes flat listings from a specified URL
• 🔍 Extracts key details: title, price, and location
• 💾 Saves data to a CSV file
• ⚙️ Supports command-line arguments for customization
• 🛠️ Easy to use and modify for different websites
• 🧪 Includes a simple test case for demonstration
• 📚 Utilizes Python’s built-in libraries along with BeautifulSoup for HTML parsing
• 🔔 Discord Webhook integration for notifications

🏢 Housing Providers

• LWB
• Lipsia
• BGL
• VLW
• Wogetra

📦 Requirements

You can run the bot natively on your machine or use a Docker image. The requirements include:

• Python 3.6 or higher
• BeautifulSoup4
• Requests
• Pandas
• Discord Webhook (optional, for notifications)
• Docker (optional, for containerization)

🛠️ Installation

1. Environment Setup

Ensure that the .env file is configured correctly. An example is available in the sample.env file. Copy it to .env and fill in the required values.
You no longer need to set SAP_SESSIONID or COOKIE_SESSION manually. The scraper opens the LWB portal with a headless browser, clicks “Ich suche eine Wohnung”, opens “Immobiliensuche” → “MEHR ANZEIGEN”, sets the maximum results to 1000, clicks “Suchen”, and extracts session cookies automatically.

2. Python Environment

You can use a virtual environment to install the necessary packages:

# Create a virtual environment
python -m venv venv

# Activate the virtual environment
# On Windows:
venv\Scripts\activate
# On macOS and Linux:
source venv/bin/activate

# Install the required packages
pip install -r requirements.txt

3. Docker Environment

Alternatively, use the Docker image provided in the repository:

# Build the Docker image
docker build -t flatscraper .

# Run the Docker container
docker run -it --rm flatscraper

🎉 Have Fun and Happy Scraping!

Wishing you a great time and speedy flat searching with the bot. If you have any questions or suggestions, feel free to open an issue on GitLab. I'll respond as soon as possible.

Happy scraping! 🚀