In the competitive realm of Search Engine Optimization (SEO), understanding your competitors is paramount. As businesses strive for online visibility, knowing who occupies the top rankings can be the key to unlocking success.
Manually monitoring competitor rankings is daunting. With the digital landscape in constant flux and search engine algorithms evolving, keeping track of competitor movements can feel like chasing a moving target.
Enter this cool SEO Python script – a powerful tool designed to simplify competitor analysis. In this article, we delve into the workings of this script, revealing how it uncovers the top contenders in your SEO arena and sheds light on their standing for your chosen keywords.
Understanding the Script:
At its core, the script harnesses the capabilities of several Python libraries: requests, BeautifulSoup, and pandas. These libraries facilitate seamless data retrieval, manipulation, and analysis.
The script’s functionality revolves around two key functions: get_domain and scrape_google. The former extracts domain names from URLs, while the latter scours Google search results for specified keywords, retrieving relevant domain information.
To gather data, the script navigates Google’s search results, extracting domains and relevant keywords with precision. Once collected, the data is parsed and organized into a structured format, laying the groundwork for insightful analysis.
Setting Up the Script:
Getting started with the script begins with the installation of necessary Python libraries. With requests, BeautifulSoup, and pandas in place, the script is primed for action.
Importing the required modules sets the stage for customization. Users can tailor the script to suit their specific needs, ensuring optimal performance and relevance to their SEO objectives.
Running the script is a straightforward process. By providing the target keywords, users can initiate the script, which then navigates the digital landscape to uncover competitor rankings.
import requests
from bs4 import BeautifulSoup
import re
from collections import defaultdict
import pandas as pd
# Function to get the domain name from a URL
def get_domain(url):
try:
domain = re.findall(r'https?://([A-Za-z_0-9.-]+).*', url)[0]
return domain
except IndexError:
return None
# Function to scrape Google search results for a keyword
def scrape_google(keyword, num_results=10):
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"}
query = '+'.join(keyword.split())
url = f"https://www.google.com/search?q={query}&num={num_results}"
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
# Update the selector according to Google's current page structure
search_results = soup.find_all('div', class_='ZINbbc xpd O9g5cc uUPGi')
domains = []
for result in search_results:
link = result.find('a', href=True)
if link:
domain = get_domain(link['href'])
if domain:
domains.append(domain)
return domains
# Read keywords from CSV
file_path = 'REPLACE ME WITH YOUR SEMRUSH EXPORT.csv' # Replace with your CSV file path
keywords_df = pd.read_csv(file_path)
keywords = keywords_df['Keyword'].tolist()
# Store domain occurrences and associated keywords
domain_keywords = defaultdict(list)
# Scrape Google for each keyword
for keyword in keywords:
domains = scrape_google(keyword)
for domain in domains:
domain_keywords[domain].append(keyword)
# Prepare data for CSV
csv_data = []
for domain, keywords in domain_keywords.items():
frequency = len(keywords)
csv_data.append({'Competitor Domain': domain, 'Keywords': ', '.join(keywords), 'Frequency': frequency})
# Convert to DataFrame and save as CSV
output_df = pd.DataFrame(csv_data)
output_df.to_csv('competitors_keywords.csv', index=False)
print("CSV file 'competitors_keywords.csv' has been created.")
Exploring the Output:
The culmination of the script’s execution is the generation of a CSV file – ‘competitors_keywords.csv’. This file serves as a repository of invaluable data, offering insights into competitor domains, associated keywords, and their frequency.
Each column within the CSV file holds a wealth of information. From Competitor Domain to Frequency, users can glean insights into competitor activity and visibility in the online sphere.
Interpreting the data requires a strategic approach. Understanding what each row represents and how to leverage it effectively is essential for deriving actionable insights.
Please note that the output is purely an example and this can/will be executed at a much larger scale with many competitors and keywords.
Benefits of Using the Script:
The script offers a plethora of benefits to users seeking to gain a competitive edge in the SEO landscape. Its automation capabilities save time and effort, streamlining competitor analysis and freeing up resources for strategic initiatives.
By providing comprehensive insights into competitor rankings, the script empowers users to make informed decisions about their SEO strategies. Whether tracking a few keywords or an extensive portfolio, its scalability ensures relevance and accuracy.
Driven by data, the script facilitates evidence-based decision-making in SEO strategy development. By leveraging competitor insights, businesses can refine their approaches and stay ahead of the curve in the ever-evolving digital landscape.
Conclusion:
In conclusion, the SEO Python script is a game-changer in the world of competitor analysis. Its ability to unveil competitor rankings and provide actionable insights is unmatched.
We encourage businesses to embrace the script and unlock the valuable insights it offers into the competitive landscape. As we look to the future, we anticipate further developments and enhancements that will cement the script’s position as an indispensable tool in SEO strategy.