• Companies will be able to extract easily their GSC data to their data lakes. It was already possible previously if you knew how to extract the GSC data, through the API, to BigQuery, but this new integration will be easier and includes more data points.
• I already explained in a previous post what the limitations of the API were, I hope this integration will solve at least a part of them
• SEO professionals will start using SQL more extensively. Some of them were already doing it, but this integration will increase the necessity of knowing at least the basics of this language. If you are worried about the cost: don’t! I have currently a database with 150M rows in BigQuery, and it costs me less than $5 per month to use it.

I’ve been using SQL for almost 3 years now, and the objective of this post is to explain the basics of SQL for SEO analysis and some simple, yet powerful, use cases. This is not an ultimate guide, and if you have advanced need you can have a look at the following courses: Codecademy, Sololearn, Dataquest.

Why should you learn SQL?

SQL is the most common language used to query modern databases. This includes simple website databases (such as the WordPress I’m using), but also more advanced data warehouses such as BigQuery. You can use it without knowing SQL through Connected Sheets, but you’ll still need to write SQL in some cases.

Apart from that, bear in mind that Excel and Sheets have limitations and if you have to handle many rows, you may face some issues. Why? Because these tools were not designed to hold millions of rows, when databases are specifically designed for that.

Other advantages of SQL that are worth mentioning:

• Easier learning curve than most programming languages such as Python. You can get started in a couple of hours.
• Unlike Excel, your logic will be easier to understand because it will be fully written. Forget the pain to reverse engineer the logic behind a full Excel file.
• You can run the same query several times (same task, different projects …) and save time to focus on tasks where you have more added value.

Do you need to learn SQL as an SEO?

No! SQL is not mandatory, but depending on the projects you work on, it can help you work better and/or faster. This should not be at the top of your hard skills list if you’re starting your training in this field.

SQL basics

To explain the SQL basics, I’ll be working with a GSC database including two dimensions (page & date) and four metrics (clicks, impressions, ctr, position). If you’ve opened a GSC dashboard before, you should be familiar with all of these. If that’s not the case, they are defined in the documentation.

A simple SQL query is usually composed of three parts:

• SELECT: the columns we want to get. You’re often interested in a limited number of columns, even if your table has a lot of them.
• FROM: the table we extract the data from. You’ll usually have a dataset composed of several tables. You need to specify the name; otherwise SQL won’t know which table you want to extract the data from.
• WHERE: the logic we apply to our data. In our example below, we just want the total of clicks after the 01/01/2022.

Simple, right? SELECT, FROM and WHERE are called SQL clauses. They have specific meanings (see full list) and must be used in a specific order. You cannot use the FROM before the SELECT for instance: SQL will throw an error.

As I mentioned, the objective of this article is not to explain everything to master SQL, but the basics you need to leverage your data. In the following sections, I’ll explain what the most important clauses are.

SELECT

The SELECT clause is a fundamental component of SQL that is used to retrieve data from one or more tables in a database. Some tips:

• You can specify a new name for your columns using the AS statement followed by the new name. I highly recommend doing that when you use aggregation methods (such as SUM() in my example) in BigQuery because otherwise you’ll end up with names such as f0_ which are confusing.

• If you want to select more than one column, you must separate the names by a comma. If for any reason you need to select all columns, use select *. Do not do that on huge tables because it may take a while and/or cost you a lot if you’re using BigQuery.

WHERE

As we already explained, the WHERE clause is used to apply conditions. You can apply conditions on columns that are not inside your SELECT clause, which is one of the advantages of SQL over Excel. It is actually what I did in my previous examples. Some comments on this clause:

• You can obviously combine conditions using the AND or OR instructions

• You can use the most common mathematical operators, such as = (equal to), != (not equal to) etc…
• To avoid creating countless conditions, you can use the IN to check if a column is equals to any of a list of provided values

• To check if a column starts, ends or contains a specific string, you can use the LIKE operator. With this operator, the percent symbol must be interpreted as “anything or nothing”.
  • wik% will check if a string starts with “wik”
  • %wik will check if a string ends with “wik”
  • %wik% will check if a string contains “wik”

GROUP BY

This clause allows you to group data. As I explained, our table includes data per URL and per date: how can we get the total by date, for instance? Using GROUP BY!

Some important details:

• This operation is similar to a pivot table in Excel.
• You cannot group data by a column you didn’t add within your SELECT clause. This would throw an error.
• If you use GROUP BY with ORDER BY and LIMIT, you can answer to questions such as “What are the X most performing pages”

CASE

The CASE statement is useful when you want to create a new column in your result based on one or several conditions. For instance, let’s imagine that you would like to understand what percentage of the traffic is generated by the homepage and compare it to the rest. You can create a simple SQL CASE to do just that:

• You can include as many conditions as you want, but you always need to follow the WHEN {conditions} THEN {value} logic
• You need to end your conditions by an END and then assign a name to your new column

In an SEO context, you will often use this clause to categorize your pages, even if there are other use cases, obviously.

JOIN

The main advantage you have when you add your data from several sources in a data warehouse, is the ability to easily merge them. You can, for instance, get the number of orphan pages but generating traffic on your site using two different data sources:

• The GSC table we already used in our previous examples
• A new table, containing the data from a crawl done with Screaming Frog or Oncrawl

In Excel, if you want to merge data, you’re very likely to use the VLOOKUP formula. Its closest equivalent in SQL is JOIN, and the most common are:

• INNER JOIN: you merge data by combining rows that have matching values in the two tables
• LEFT JOIN: you merge data by keeping all data from the left table. When you use VLOOKUP in Excel, you usually perform an operation similar to this kind of JOIN.
• FULL OUTER JOIN: you merge data by keeping data from both tables

If you’re not familiar with these concepts, the following image (source) should help you understand:

Going back to our gsc and screaming_frog tables, we can for instance use the JOIN statement to get the number of clicks by depth using the following logic:

• The clicks will come from the gsc table
• The depth will come from the screaming_frog table
• These two tables are merged using the page column

The code we’d use will be the following:

select 
--the data we need 
--the column name starts with the name of the table followed by a dot because we're using more than one 
--It is not mandatory if you don't have columns with the same names in your tables
SUM(gsc.clicks) as clicks,
screaming_frog.depth
--our left table 
from gsc
--our right table and the join logic 
left join screaming_frog on gsc.page = screaming_frog.URL
--rest of the SQL code
group by depth
order by depth asc

This code would return the following table. Note that we have a row with no depth information: these are the URLs found in the gsc table, but not in screaming frog. If we had performed an INNER or RIGHT join, this row would not be here.

Conclusion

Is that it? Of course not, but if you master these concepts, you’ll be able to start using SQL. Other concepts, such as subqueries, can be learned along the way when you’ve mastered the basics.

Again, SQL is not a mandatory hard skill to work in SEO, but it will definitely improve the way you work if you have to deal with a lot of data for your clients or projects.

The post SQL for SEO appeared first on Antoine Eripret.

That being said, there are situations where search volumes are useful and accurate. It’s easy to get the data for a couple of hundreds with any mainstream tool or the Google Keyword Planner if you have running ads (otherwise you’ll get volume brackets). But how can you retrieve search volumes for thousands or even millions of keywords? In this article, I’ll go through my go-to solution.

Why would you even need data for thousands of keywords?

This is a legit question you may ask yourself. At the end of the day, we usually don’t need to get the search volume for every keyword we have in our Google Search Console, for instance. That’s true!

But imagine for a moment that you work for the French national railway company (SNCF). The French network is composed of more than 3,000 train stations (source), which ends up creating a bit less than 9,000,000 possible routes between them. You’ll obviously want to cover the most searched in your architecture, and you’ll have to base your decision on:

• First-part data such as internal searches and sales
• Third-part data, such as the number of searches on Google and other search engines

And this is where the ability to retrieve search volume at scale will be handy!

Usual approach

If you want to get data for your keywords, you usually have two options.

Google Keyword Planner

This tool from Google can be the easiest option in most cases. Nevertheless, you cannot add more than 700 keywords at once. If you have a couple of thousands, you could add them by batch, but it quickly becomes annoying and time-consuming.

There is still the Google Ads API, but setting it up is complex and requires making your way through Google’s documentation.

Other tools

Some tools have an API to get keyword data, including search volumes. But I don’t often recommend using most of them for many reasons:

• Price: the API is (almost) always only available with the most expansive plans. Moreover, you often have to spend a lot of API credits to get all the information you need.
• Documentation: the documentation (even if Semrush has improved a lot on that) is bad. Have a look at what ScrapingBee does (source) where everything is well-explained, and code snippets can be copied. And then compare it to most SEO providers. I really don’t like spending an hour figuring out how the API works.

• Batch requests: one critical features for our usage is to be able to retrieve data by batch, meaning that one call can return the data of hundreds or thousands of keywords. Most of the API I’ve tried don’t allow it.

How do I do it?

Obviously, I’m not writing this article to let you down and tell you that you can’t retrieve keyword volume at scale. I’ve discovered an API (DataForSEO) that has all the required functionalities you need. DISCLAIMER: I’m not paid by the company to write this article. I just happen to be a delighted customer, but they are not well-known.

The only issue: it’s an API; therefore you need to write code or use a no-code tool to do it for you. But you can get the volume for 1,000,000 keywords in less than an hour for just $50. Amazing, right? Also, you can get data for Google but also Bing.

I won’t provide the codes you need to make the calls because:

• I work in Python, but you may use another language. If you are also using Python, have a look at this repository to speed your implementation. If you’re not, have a look at the documentation.
• This article is aimed at explaining the steps you have to follow. If you need help with the code, reach out to DataForSeo.

To get the search volume, you need to follow these steps:

• Pick a location: you have a specific endpoint to get the list. These locations are the same available in Google Ads and are usually a country, a region or a city. Once you’ve found yours, write it down. You can also use the location_name, but I prefer to use the latter.

• Divide your keywords in batch of up to 1000: One of the best features of this API is that you just need one call to get the data for 1,000 keywords. If you have 1,600 keywords, you need to send two calls: one with 1,000 keywords and another with 600. Using the POST method (see documentation) you then request the data. The data is not sent immediately unless you use the LIVE method (more expensive). Be careful to check the limitations you have for your keywords (forbidden characters, size) because if one of your keywords doesn’t pass the check, the full batch will fail.

• Wait a moment and check that your data is available: based on the number of batches, you may have to wait <1mn or way more. But there is an endpoint to check if your data can be downloaded.
• Download your data: using the last endpoint, you can download what you wanted. The output (after a bit of data manipulation): the data you need!

Conclusion

Retrieving keyword volume at scale is sometimes required, as it helps you to understand the popularity of specific keywords. The method mentioned in this article is the one I use, and I always recommend. If you don’t know how to code, just ask someone who does and save time and money on the long term!

The post How to retrieve search volume at scale? appeared first on Antoine Eripret.

As a summary, trying to SEO a JavaScript-powered website usually means that you need to pay attention to two things:

• Ensure that search engines and users can see the same things. To achieve this objective, you’ll often rely on SSR (Server-Side Rendering) as CSR (Client-Side Rendering) can lead to longer rendering queue.
• Ensure that key SEO components, such as links for instance, are not manipulated during the rendering process

Sounds straight-forward, but it is not always as easy as it sounds. Moreover, there is a specific issue that is often caused by JavaScript frameworks that I would like to explain today. Because it is not well-documented and can potentially kill your SEO. And is linked to what is called content rehydration.

What is content rehydration?

Content rehydration is a process that occurs when a website, built with a JavaScript framework, such as Angular or React, dynamically updates the content on a page without requiring a full-page refresh. This means that when a user navigates to a new page, the core structure of the page is loaded first, and then the dynamic content is loaded in real-time, improving the loading speed and user experience. This process is called rehydration because it “brings the content back to life” after it has initially been loaded. In Angular, this is achieved through the use of server-side rendering and client-side rendering techniques, which work together to provide a fast and seamless experience for the user.

Why using rehydration instead of relying only on SSR? Simply because it cuts the number of operations you require your server to do before sending the response to the user, while ensuring that the application is interactive quickly.

If you ensure that the main HTML components are included in your raw response sent by the server, we should be fine, shouldn’t we? Well, it’s not that simple. Let me tell you why.

What is the issue with content rehydration?

By default, this behavior comes with a major flaw, though.

It will add a script to the raw response sent by your server with all the required code to make the application dynamic. Out-of-the-box, this script can easily represent more than 90% of the total HTML size, as the example shown below from this tool. Nevertheless, 1.07 MB (270kb compressed) is not something that should have that much impact, isn’t it?

I used to believe that. Boy, little did I know how wrong I was.

The issue is not linked to the size of the script itself, but what it means for search engines.

Google explains clearly in its documentation how it parses HTML document. The bottom-line is that once the raw response has been received, JavaScript files are executed to render the final HTML. The more JS code you have, the longer you’ll have to wait for your page to move from crawled to indexed.

If you think from Google’s perspective, it makes complete sense. Indeed, this behavior rewards websites relying heavily on SSR, which decreases Google’s operating costs because parsing JavaScript at scale must be expensive. If you use out-of-the-box content rehydration, your website will take time to be indexed because every page will spend more time waiting in the rendering queue.

What impact can it have during a migration?

Scary, but what if I told you that there is a situation where it could actually be worse than that? Imagine that you are migrating from an old website to a Javascript-powered where you need to change your URL structure. Not an uncommon situation if you think about it.

Well, you’ll face the following situation in countless keywords: a sudden (but temporary) drop followed by a new position that is very likely to be lower than the previous one, at least in the short term.

Why you may ask? Let me explain:

• During the phase 1 (in my previous screenshot), we observe changes that are completely normal, as a result page can evolve daily
• During the phase 2, a couple of days after the redirection is implemented, the position suddenly drops.

The strange thing about this phase is that the ranked URL is neither the old URL nor the new one. It’s another one, usually not matching the user intent. My initial reaction was simple: Google is broken and is not handling redirections as well as it used to.

Easier to blame a third party when the issue actually comes from your side, right? What was happening was actually simple:

• Google detected the 301 redirection and added the new URL in its crawl queue
• Due to the content rehydration on the new pages, Google took forever to process the content
• At some point, it removed the old URL from its index (respecting the 301) but as the new URL was not processed yet, it picked the closest matching URL for a while
• When the new URL is finally processed, the new URL got back to (almost) the old rank

The difference for the position between phase 2 & 3 is explained by the difference in speed. While CWV are a tie-breaker, if your new page is significantly slower than the old one (content hydration will impact all CWV metrics but CLS) and given the resources Google will spend on your page, you’ll very likely not recover your initial rank.

How can you solve it?

Content rehydration is not bad because it comes with major advantages I already explained at the beginning of the article. Nevertheless, out-of-the-box, the script includes a lot of information you are not using in your application. The trick is then to optimize it by asking your team to exclude the information you are not using.

Too simple to be true? Well, that’s precisely what was done here and after the initial drop, script size was reduced by 50-75% and performance level went back to the initial level instantly. And rankings started to increase just after that.

Conclusion

While having a huge script doesn’t necessarily affect only JavaScript-powered website, I’ve realized that this issue affects a lot of them, even it is often not detected until a migration happens, At the end of the day, we often look at the total HTML size, but not its breakdown by tag.

If you’re dealing with a JavaScript-powered website, make sure you know what content rehydration is and explain to your team how it needs to be optimized to avoid breaking your SEO.

The post What is content (re)hydration, and how can it impact your SEO? appeared first on Antoine Eripret.

They indeed leverage what is called programmatic SEO, a smart way to scale your traffic. In this article, I’ll explain what you need to know about this concept that, even if you’re not familiar with the name, you already know.

What is programmatic SEO?

Programmatic SEO allows you to generate many pages using first or third-part data to generate the content. The definition is pretty straightforward. Let’s see one example with the project I created. If you’re curious, the URL is https://comoescribir.net/.

The website, in Spanish, helps you know how to correctly write numbers from 0 to 50,000. As you may have guessed, even if I managed to generate some traffic, this is not a project I could live from (I’m actually making around €50/month with it), so feel free to copy it if you like. The main template, used by 50,000 URL, leverages python libraries to provide the spelling for main numbers in several languages.

One myth I often read, and I’d like to debunk straight away: programmatic SEO is not only designed for ranking on long-tail keywords. Big websites like Amazon, Skyscanner or Booking, to name just a few, rely extensively on programmatic SEO to build their architecture and generate traffic at scale. And let me tell you that these websites are not going after long-tail keywords

How can I leverage programmatic SEO?

Now that we’ve agreed on what programmatic SEO is, how can you use this information for your project? The theory is actually simple, but let’s use a real example Let’s assume that we are working for a train company in the UK and let me guide through the process.

Find the head term

Our objective is obviously to rank on keywords searches by users looking for a train ticket between two cities. You don’t have to be a genius to deduce that the main keyword will be “train london manchester” where London and Manchester can be replaced by any city. Even if you have variations for the same intent, it seems that this keyword is the most searched one. And very likely to convert well.

Map the demand

We have the search volume for one option, but we obviously want to map the full demand. In our example, it means:

• Getting our hand on a complete list of UK cities. With more than 2,600 cities, we could use this database for instance. Depending on the information you are looking for, you may have to build the database yourself, though. Not everything can be scraped.
• Build the keywords: with 2,600 cities in our database, we’d end up with more than 6M keywords. You don’t always have to handle that level of data, but be prepared to be surprise sometimes.
• Extract the volume: getting the search volume for more than 6M keyword is obviously not something you want to do manually. You can do it quickly and in a cost-efficient way using https://dataforseo.com/. If I have more than 1,000 keywords, that’s the API I always end up using.

At this stage, you’ll have a clear view of the demand and the 6M potential routes may be reduced to a couple of dozen of thousands. You may narrow down the list more by using first-party data like sales, obviously.

Create a template

If we look at a competitor, Trainline, we can see that they have pages equivalent to what we want to build.

Mapping the demand is easy, but now we need to build the pages. This means adding data regarding:

• Prices
• Information about the journey
• Information about the cities
• Tricks to find cheap tickets
• etc….

It can get complex pretty fast, and you have to remind that the goal is to scale page creation; therefore you must avoid, as much as you can, manual content creation. Because it would be a nightmare to update and/or translate into another language. But even without manual content creation, these projects can get complex quick and involve several teams (SEO, Data, Back-end and Front-end).

Think about the internal linking

Creating these pages you want to rank is not enough. In our example, we may end up with thousands of URLs, how are we going to feed them to Google? You have to find a relevant way to do so.

• Usually, the solution is to create an additional level in your architecture, not aimed at generating traffic but crucial for the internal linking
• You can also leverage main navigation and links between pages

For the project I mentioned at the beginning of the article, the internal linking mainly depends on links between pages, which are built using a ghostblock-like strategy.

Conclusion

Programmatic SEO is, in my opinion, one fancy term to explain what has been used for decades by companies to create & rank pages at scale. Amazon or Walmart, to name just a few, are using this approach. Because it is necessary if you work on a project targeting a very heterogeneous demand.

The only question is: is it relevant for your project, and do you have the necessary data to achieve this approach?

The post Programmatic SEO: How to scale your traffic? appeared first on Antoine Eripret.

That being said, the data provided are often incomplete or incoherent. And it’s important to understand the data you are using as a basis for your recommendations, to avoid taking bad decisions.

UI vs API

Most of you are surely familiar with the UI. You may also know that for big projects, you shouldn’t rely too much on it. Several issues:

• The UI won’t provide more than 1,000 rows, no matter what tab you are looking at. For instance, I have a property with more than 50,000 active URLs, and yet just 1,000 are shown. There is no workaround because even if you download the data using the “Export” button, you’ll be left with the same amount of data.

• Filters have been improved since RegEx are supported, but it’s still impossible to apply more than one filter to the same dimension.
• You can’t resample data per week or month. I usually compare weekly or monthly data. Daily data are impractical.
• You can’t display results for combined dimensions. For instance, you can’t export a table including top URLs per country. The only way to achieve that using the UI would be to filter by country (one by one!) and export data individually.

So yes, the UI is easy to access, but you must be aware of the limitations. The API removes almost of all these limitations, and you can choose the best option to use it based on your knowledge and workflow:

• Google Sheets: install this add-on and export your data directly into Google Sheets. A huge time-saver. Please note that you won’t be able to handle huge datasets, though, due to Google Sheets limitations.
• Python: Use this library, which is simple and does the job perfectly. If you’ve never used it before, be sure to follow this tutorial on how to set it up. I’m sure there are equivalents in other languages, but I cannot guide you as I’m only using Python.
• Manual setup: You can also create your own program to access the API. Just head over to the official documentation and try to follow the steps.

All dimensions are not equal

If you’re familiar with GSC, you already know that you have five main dimensions: date, page, device, country & query, The names are self-explanatory, and I don’t need to explain what they represent.

If you query one of these dimensions, and even if the API returns more than 1,000 rows, the total number of clicks & impressions can vary. For instance, when I included the query dimension, I ended up with only 66% of the clicks and 29% of the impressions. The (main) official answer for these discrepancies are anonymized queries. Not quite sure about this one, but even with this percentage, Google Search Console is still your best keyword database. Please note that this percentage may be higher or lower in your case, as highlighted by a recent study done by Ahrefs.

You can also be surprised by why I have 120% of my impressions when I include the page dimension. The explanation is actually simple: more than one URL can be displayed for a given query (especially on your branded terms) and hence inflate impressions.

Bottom line: do not forget that when you look at data grouped by query, you are actually looking at a sample. The rest seems accurate.

Now, what would happen if we combine two dimensions? I explained before that one of the main advantage of the API was to be able to combine dimensions. Does it impact the number of clicks and impressions returned as well? I created all possible combination and I ended up with the following results (see table below).

There are some obvious and not so obvious conclusions from this extraction.

• The order of the dimensions doesn’t matter. The results are the same if you retrieve the data for date/page or page/date. Might be obvious, but it has to be tested anyway.
• The second conclusion is the most counter-intuitive. When we retrieved data grouped by query, we got 66% (clicks) and 29% (impressions). When you add the page dimension, these values are increased to respectively 73% and 86%. This makes no sense. The jump for the impressions can be explained by queries displaying more than one URL, but the jump for clicks has no logical explanation.

Using properties for your folders makes sense

When Google announced the URL inspection API with a daily quota, one of the workarounds was to create one property by folder to increase it. Out of curiosity, I wanted to know if there was a discrepancy for clicks & impressions for:

• The main property, keeping only data for a specific folder
• The folder property

There is indeed! While the number of clicks is the same, the impressions are slightly higher for the folder property. I’d say that the folder property is more likely to return more data because the dataset is smaller. If you need to calculate CTR, you better use folder property. Because in this example, it goes from 4.3% to 3.9%.

Be careful with the average position

The average position in Google Search Console is often used to assess if a project is improving. It is sometimes useful: in the screenshot below, you can see that there is a correlation between the improvement in the average position and the CTR.

That being said, the average position is something I’d ignore, for several reasons:

• In a world where Google is adding more and more element in its SERP (from ads to modules), your CTR depends more on your pixel position than anything else. Therefore, the position is almost irrelevant
• Your average position can actually decrease if your results improve. Imagine that you start to rank 9th on a generic query: your impressions would go up, but your position may go down.
• In some industries, Google displays a module highlighting results from comparators & classified. Awesome, right? Well, not that much because any website including in this module will have a reported average position of 1 even if the CTR from this module is super low. Which, by the way, prevent you from building a relevant CTR yield curve with your data.

Conclusion

As we’ve seen, there are some details that you better understand when you use Google Search Console to prevent you from taking bad decisions. Does it mean that data are bad? Not at all! As I highlighted in my introduction, Google Search Console is still one of the best (and free) SEO tools that you can use to improve your results. You just need to know how to interpret the data you are looking at .

The post What you’re not told about GSC data appeared first on Antoine Eripret.

Which I honestly believe because I’m yet to meet someone who would claim that. But you know what? It doesn’t matter. I use RegEx daily, and yet I master only the basics. Which is more than enough for most cases.

Why and when should I care about RegEx?

There are many skills that you can learn and apply in the search industry. Their usefulness depends on several factors, such as:

• Your seniority level: I wouldn’t advise learning RegEx if you don’t master the basics of SEO.
• The kind of SEO you are: RegEx are more useful when you deal with a lot of data. If you make a living by creating niche websites, it may not be as useful as if you’re responsible for the SEO strategy at Walmart.

Don’t learn it just because you’ve liked this post, learn it because it could help you during your daily tasks.

That being said, keep in mind that RegEx are supported by the vast majority of SEO tools such as Google Analytics, Google Search Console, Screaming Frog, Oncrawl and I could go on forever. You even have Google Sheets formulas (REGEXMATCH and REGEXEXTRACT) to use them directly in Excel.

One smart example to leverage RegEx inside Google Search Console to pull questions around your brand. And I’ve used this trick in the past and it’s awesome to improve your FAQ / Customer support section. Without RegEx, pulling these questions would take more time.

RegEx 101

As I already highlighted, in most cases, RegEx can actually be pretty simple. If you understand a few concepts, you’ll be able to use them with, maybe, a bit of Googling.

dot (.)

The dot is a wildcard that can match any character. It can be a letter, a number or a special character, it doesn’t matter. It’s not often used by itself, though: it usually comes with a quantifier that I’ll explain later in this article.

circumflex accent (^)

The circumflex accent indicates the beginning of a text we are trying to match. ^apple only matches apple in my example below because the other word starts with the letter m. In plain English, this character will allow identifying strings that start with a specific text.

dollar sign ($)

The dollar sign does the opposite of the circumflex accent: it allows identifying a string that ends with a specific text,

backslash (\)

As we are explaining, some characters have a special meaning when you are using them in a RegEx context. Now, what about matching these characters and not their meanings? How can we match a text including circumflex accents, for instance? You just have to use the backslash before them, whose purpose is to indicate that this character should be interpreted literally, and not with its meaning.

several options

Sometimes, you can have ore than one option to match a string. RegEx have several patterns to achieve just that:

• [abc] indicates that you can have the letter a, b or c. Any of these options is fine. Example:

• [a-z] matches any lowercase letter from the Latin alphabet (if you speak spanish, ñ is not included).
• [A-Z] does the same, but for capital letters
• [a-zA-Z] matches any letter, whether it is lowercase or capital
• [0-9] matches any digit

It’s not that hard, is it? The only thing you’ll have to remember is how to type a bracket using your keyboard. Because it will be required quite often.

Quantifier

While the regular expressions we’ve seen in the previous sections will cover most of your cases, we still need to learn a key concept: the quantifier. These characters indicate how many times a specific string can be repeated.

A quantifier is always applied to the character located before. In my example below, the plus sign (+), which mean 1 or more times, is applied to the letter o. Which is why it matches with the two first options, but not the third because it doesn’t include the letter o.

The main quantifiers are the following:

For instance, we can match two folders, /flight/ and /flights/, with a single RegEx by indicating that the letter s is optional.

Let’s practice: a simple example

Let’s apply some of the concepts we’ve seen so far. Let’s imagine that you are working for a website selling plane tickets. Your web architecture includes thousands of URLs built following this pattern:

• URL: https://www.example.com/flights-barcelona–paris_bar–par.html

• Explanation: https://www.example.com/flights-{origin}-{destination}_{airport_origin}-{airport_destination}.html

From the explanation, we need to transform the variable into valid RegEx:

• origin / destination: a word without space: [a-z]+
• airport: three underscore letters: [a-z]{3}

Which will become:

https://www.example.com/flights-[a-z]+–[a-z]+_[a-z]{3}–[a-z]{3}.html

That’s completely normal if it’s not (yet) easy to come up with custom RegEx quickly, but you’ll get there. As I highlighted, you may encounter specific cases where the RegEx I explained in this article are not enough, but you should be good for most cases.

Let’s practice with a more complex example

For this second example, let’s assume that we are working for Animalear, a Spanish E-commerce for pets. I’ve actually worked for this client when I started my professional career in Spain. It brings good memories back

We know that this client has a problem with some of its product URL.

• Some are user-friendly such as https://animalear.com/perros/scalibor/collar-scalibor-antiparasitario
• Other aren’t such as https://animalear.com/perros/nutro/p-195619

You can obviously rank with both, but the former is better for several reasons. What we want to do is simple: check that we are not using the bad structure in our internal linking from key pages such as the homepage and categories.

We could obviously perform this check manually, but if we have more than 10 pages, we’ll spend a significant amount of time. Let’s see how we can use RegEx to help us here.

• Let’s crawl the full website using Screaming Frog or any similar crawler

• We then export the inlinks report

• We then open the file in Google Sheets. Please note that if the crawl is too big, you may not be able to open the file in Sheets.

• We continue with our process: we are looking for links FROM categories TO non-optimized URLs. Categories have two different structures: https://animalear.com/pajaros/c_juguetes-para-pajaros and https://animalear.com/gatos/s_accesorios-casetas. In plain English: domain + a folder + the letter c or s followed by an underscore and the rest of the URL. Which can be translated in REGEX; using the explanations given before, by https:\/\/animalear.com\/.*\/[cs]_.*
• Applying the same logic with product URLs (such as https://animalear.com/perros/nutro/p-195619), we define the RegEx https:\/\/animalear.com/.*/p-[0-9]+$. In English: domain + folders + the letter p + dash + some numbers
• Using the REGEXMATCH formula, we can filter our file to include only rows with the links we were looking for.

Quicker than doing the whole thing manually, right? And we didn’t create complex RegEx.

Conclusion

As I mentioned in my introduction, RegEx can be complex, but they often won’t be. By knowing your way around the basics, you’d be able to:

• Work smarter in Google Sheets
• Leverage Custom Search & Extraction in Screaming Frog or Oncrawl
• Categorize your sections in Data Studio
• And so much more

I strongly advise taking some time to learn the basics if you are dealing with a lot of data. You don’t have to know everything, and you will save so much time in the long term that it will be worth it.

The post RegEx (regular expressions) for SEO appeared first on Antoine Eripret.

If handled poorly, a web migration can have a tremendous effect on online visibility and traffic. Which is obviously what you don’t desire to live through. That being said, a poorly handled migration is, unfortunately, one of the main reasons a business owner will start to take SEO seriously. When I was working agency-side, some of our clients started to work with us after a horrendous experience.

In such a situation, your number one priority when you onboard the client is to understand why the migration is having such an impact on traffic generation and how to fix it quickly. Often, redirects have been implemented poorly, and you need to start reviewing them. A very time-consuming, that can take, literally, days depending on the size of the project. This task is often handled to junior team members (I’ve been there) whose only purpose during days is to fill in an Excel file. 

There are situations where it is the only situation because you cannot find any pattern to speed up your work. Fine, but often there are indeed better ways to proceed. 

Let me walk through two quick and cost-efficient approaches you can take to save days of manual work to your team or yourself. 

The context 

To provide a real context to the article I’m writing, we’ll use the migration that happened some months ago for the French railway company. They decided to move from https://oui.sncf/ to https://www.sncf-connect.com/. 

The migration was actually handled correctly from an SEO perspective, but let’s assume that it wasn’t and that the company comes to us to try to fix it. They literally have thousands of URLs, so mapping them manually is not an option.  

If you look at the position history, you’ll realize that the path (the URL stripped from the domain name) is actually the same, so we could handle this case with a simple redirect rule. But let’s assume – again, I know – that it’s not the case and that we have to find the equivalent, at scale. 

Option 1: use semantic proximity 

A web migration often comes with some URL changes (which is why you need redirects), but you frequently keep a similar structure. You may add some new words, remove some numbers, but there is a common ground. 

This option leverages this and try to find the closest equivalent between our old & new structures, by looking at the URL only. Why don’t we look at content? 

• Old content may not be available anymore 
• Crawling a website takes time and if we can skip this step, we’ll be able to help our client faster 

Now, how the heck are we going to do it? 

Fuzzy matching 

If you don’t know what fuzzy matching is, you can read this amazing piece by Lazarina Stoy, who explains everything. 

As a summary: 

• Fuzzy matching allows us to match similar strings. This is what Google uses to correct spelling mistakes, by the way: 

• There are several algorithms available but in our case, we’d use the Levenshtein distance (match words based on number of edits necessary to get from one word to the other) or TF-IDF which looks at how often a set of strings are repeated. 

I often use directly Polyfuzz (python library) when I need to use fuzzy matching because it’s effortless to set up. Do not worry if you don’t need either Python or this library, I’ll explain everything and even provide a template. 

Our logic 

The logic we’ll code is actually pretty simple: 

1. We’ll first retrieve the two URLs lists we want to match (the old and the new one) 
2. Run the fuzzy matching between them to define our redirects 
3. Compare the output of our approach with the actual redirections the SNCF has in place to calculate the success rate 

We’ll try to match around 3,000 old URLs to a potential list of around 30,000. I’ve intentionally used a smaller number for the old URLs because the second option (that we’ll see in the next section) is not free, and I didn’t want to spend a lot of money on this explanation. 

The code

You can find the code and output here. Even if you don’t understand Python, you’ll see that it’s pretty straightforward. 

We first the libraries we need for the script: pandas and polyfuzz. 

#load libraries
import pandas as pd
!pip install polyfuzz
from polyfuzz import PolyFuzz

We then load our URLs lists (stored in an Excel file) in pandas DataFrames (which is an equivalent of a table in Python):

#load urls lists
old = pd.read_excel('/content/drive/MyDrive/Website/Content/data_for_posts/redirects_at_scale.xlsx', sheet_name='old')
new = pd.read_excel('/content/drive/MyDrive/Website/Content/data_for_posts/redirects_at_scale.xlsx', sheet_name='new')

We continue by converting our pandas DataFrames to regular Python lists (otherwise polyfuzz won’t work) and we launch the fuzzy matching.

#convert to Python list (required by Polyfuzz)
old = old['URL'].tolist()
new = new['URL'].tolist()

#launch fuzzy matching
model = PolyFuzz("TF-IDF")
model.match(old, new)

#load results
result = model.get_matches()

The output: our redirect mapping. The output is almost perfect (only 1.3% error our of almost 3,000 URLs) but please note that we are in an ideal situation here, where the old & new structure are very alike.

But still, this approach would save you a ton of time and would only require a manual check before its implementation. And you can generate it in less than 2 minutes. 

Option 2: hey, Google! Do the work for me! 

The first approach is my favorite, but its output depends heavily on how close your URLs structures are. Likely, but not always the case. 

The second option is more robust, but only works if the new content is indexed by Google, which means that you’ve been contacted only some days after the migration took place (and it may be too late). Also, it’s not free, although it’s not expansive either. 

Our logic 

Again, the logic is simple: 

1. We’ll first, using first or third-part data, extract the keyword that was generating the most traffic per URL
2. We’ll then, using the site: operator, get the most relevant result, according to Google, for this query 

For instance, https://www.theguardian.com/football/world-cup-2022 is the most relevant URL for the keyword “world cup” for The Guardian. 

To retrieve data at scale, we’ll use ValueSERP, a great SERP API. 

The code

You can find the full code here. This one is more complex than the previous one, and I’ll try to explain it fully. 

The first part, as always, is intended as loading the libraries we’ll need for our script: pandas, requests and json. We also add a variable including our API key from ValueSERP. It can be found at the top of your dashboard.  

#load libraries
import pandas as pd
import requests
import json

#valueserp key
api_key = ‘’

We continue by loading the data from Semrush from our old website. In my example, I use third-part data, but if you can, I strongly advise using data from GSC. The logic being to end up with the most important keyword (in terms of traffic) by URL. 

#load file containing keywords
kw = (
   pd
   .read_csv('/content/drive/MyDrive/Website/Content/data_for_posts/oui.sncf_keywords.csv')
   #keep only best kw by traffic
   .sort_values(by='Traffic', ascending=False)
   .drop_duplicates('URL', keep='first')
   #add column with keyword to use in Google
   .assign(Keyword_Google = lambda df:'site:sncf-connect.com '+df.Keyword)
)

#remove useless columns
kw = kw[['Keyword','URL','Keyword_Google']]

At this stage, we have a DataFrame containing the Google query that we’ll use to find the redirect per URL.

We now can send the information to ValueSERP. I won’t explain in-depth this part because it is based on the documentation, but if you want to use my code, you’d need: 

• To create more than one batch, because you cannot have more than 1,000 queries per batch. I decided to limit my extraction at 1,000 to reduce the cost associated to this article 
• Update the location, google domain, device …. based on your project 

#list of queries to send to ValueSERP
#in this example, I'm just getting the data for 1000 keywords
kws = kw['Keyword_Google'].head(1000).tolist()

location = 'France'
google_domain = 'google.fr'
gl = 'fr'
hl = 'fr'
device = 'desktop'
num = 20
      
param_list = []

#create a list of parameters for each set
for i in range(0, len(kws)):
 param_list.append({
 'api_key': api_key,
 'q': kws[i],
 'location': location,
 'google_domain': google_domain,
 'gl':gl,
 'hl':hl,
 'device':device,
 'num': str(int(num))
    
 })
      

#create our batch
body = {
"name":'Demo_SNCF_Connect',
"enabled": True,
"schedule_type": "manual",
"priority": "normal",
"searches_type":"web"
}

#create batch
api_result = requests.post(f'https://api.valueserp.com/batches?api_key={api_key}', json=body)
api_response = api_result.json()
#get id
batch_id  = api_response['batch']['id']

#send data to batch
for i in range(0, len(param_list)):
   body = {"searches":[]}
   for param in param_list:
       body["searches"].append(param)

api_result = requests.put(f'https://api.valueserp.com/batches/{batch_id}?api_key={api_key}', json=body).json()

Once this code is executed, you’ll have a batch created in ValueSERP and you can (manually) launch the extraction:

It is usually superfast (<1mn) and once it has finished, you can copy the batch ID to execute the rest of the code, which basically retrieve the information provided by ValueSERP, apply some filtering to get the desired data.

#required parameters
params = {
 'api_key': api_key,
 'page_size':1000
}
batch_id = ''

#get results for ou batches
results = pd.DataFrame()
#get batch info
api_result = requests.get(f'https://api.valueserp.com/batches/{batch_id}/results/1/csv', params=params)
for url_csv in api_result.json()['result']['download_links']['pages']:
 results = results.append(pd.read_csv(url_csv), ignore_index=True)

#get results for ou batches
results = pd.DataFrame()

#get batch info
api_result = requests.get(f'https://api.valueserp.com/batches/3F3A94E1/results/1/csv', params=params)
for url_csv in api_result.json()['result']['download_links']['pages']:
 results = results.append(pd.read_csv(url_csv), ignore_index=True)

#keep only top URL
results_filtered = results[results['result.organic_results.position']==1]
#remove unsucessful scrape
results_filtered = results_filtered[results_filtered['success']==True]
#keep only useful columns
results_filtered = results_filtered[['search.q','result.organic_results.link']]
#use more standard names
results_filtered.columns = ['Keyword_Google','Redirect_URL']
results_filtered.head()

We end up with a DataFrame including our query and the best result, according to Google. We can now merge this data to our initial table, like we did when we were using Polyfuzz.

The output is actually worse than our previous solution (8.6% of error) but it is still a quite satisfactory result. Imagine the number of hours saved anyway! 

Conclusion

Redirections can be time-consuming, but if you use any of these two options to speed up your work, you can deliver more value to your client in a matter of hours than a full team doing manual work in days. And you can then spend more time on tasks that cannot be automatized! 

Work smarter, not harder  

The post Migration nightmare: how to define redirects at scale? appeared first on Antoine Eripret.

Great, but what if your need is not covered by an existing extension? In today’s post, I want to explain how to create & publish a Chrome extension from scratch from a newbie (myself) point of view.

The idea 

As most SEO professionals, I’m often using Google Search Console for my projects. When it comes to keywords data and even if the query report is kind of sampled, the tool gives us the most comprehensive data we can find. Nevertheless, when I’m performing keyword research for new content ideas or to improve my existing contents, the typical process is the following:

• Analyze data from GSC UI or using the API 
• Extract search volume using a third party tool (Google Ads, Semrush….)  
• Merge everything using Excel or Sheets

But isn’t there a way of displaying the search volume directly within the GSC API? It would cut down the time spent on this task significantly because all data would be unified in the same UI. 

When I realized that no other extension (that I knew of) offered this functionality, I decided to build my own. 

How does a Chrome extension work? 

If you don’t know it yet, Chrome extensions rely heavily on JavaScript. Based on the complexity of the one you want to create, you may need several files to make work, but you need at least: 

1. manifest.json: a file including several key information on the extension, like its name, its description and when it should be executed once installed. For more information, please see the official documentation. 
2. code.js: the name can be different (you specify it in your manifest.json file), but this JavaScript file includes the code to be executed by your extension. In my case, all the instructions are there. 

If you are curious to know how your favorite extensions work, you can follow this tutorial. 

How does my extension work? 

If you use the extension, you can do two things: 

• Display search volume directly in GSC UI 
• Add a new download button to download these volumes (if you want) along with your keywords data 

The full code is available in my GitHub repository, but let me explain how it works piece by piece.

JavaScript vs Python 

If you’ve read other articles I’ve published on my blog, you may know that I’m regularly using Python to speed-up my work. In other words: JavaScript is not my thing. Therefore, how the heck did I create an extension based on this language?

• Collaboration: the SEO industry is great. If you ask for help, there is always someone willing to help you. In my case, I spoke with Jose Luis Hernando who’s more than knowledgeable on JS and he helped me a lot through the project. I also asked Fede Gomez to solve a specific issue I had at the end of the project, which I will explain later. 
• Knowledge transfer: even if JavaScript has some key differences with Python, some concepts are basically the same. You just have to Google quickly how to write a loop, but you don’t have to understand the logic because you already learned it before. 

The Logic 

The logic followed by the extension is pretty simple:

1. Download keywords from GSC UI
2. Extract their volumes from an API 
3. Add them into the UI 

Download keywords from GSC 

function get_keywords() {
  //array where we will store our keyword
  let arr = [];
  //gets table
  //there are several tables with the same class
  var oTable = document.getElementsByClassName('i3WFpf')[0];
  //gets rows of table
  var rowLength = oTable.rows.length;
  //loops through rows
  for (i = 0; i < rowLength; i++) {
    //gets cells of current row
    var oCells = oTable.rows.item(i).cells;
    //loops through each cell in current row{
    arr.push(oCells.item(0).innerText);
  }
  // Remove "Top Queries" header from array
  arr.shift();
  return arr;
}

The data we want to extract is included in a table whose class is “i3WFpf”. I really hope that Google won’t change that soon, otherwise my extension will need to be updated!

To extract the keywords, I loop through this table and extract the first row (the keyword). I end up with an array (a list) of up to 1.000 keywords, which is GSC’s UI limit. This step was pretty easy because I honestly believed that the source code was going to be messier, like in Amazon for instance. Fortunately, that was not the case at all.

Extract their volumes from an API

Before even explaining the code, I want to highlight which API I used and why. My extension uses Keyword Surfer’s (that you may know for their Keyword Surfer extension) because:

• It is free
• It doesn’t require neither authentification nor API key (at least now)
• It allows bulk request: with one request, you can retrieve the volume for up to 50 keywords.

The last advantage was decisive, because having to execute up to 1.000 requests (for up to 1.000 keywords) would have impacted A LOT the speed at which the extension can retrieve search volumes. It also means that I won’t surcharge their API with my extension, which is the last thing I want.

That being said, the API is far from perfect:

• Some volumes are not updated very often
• Low-volume keywords are badly reported
• As usual, keywords are grouped. “Buy car” and “buy cars” will return the same volume (or one of them will return 0) when we do have separate metrics in GSC. This is a very known limitation for most API though, not only Keyword Surfer’s.
• If the Keyword Surfer doesn’t have data for a specific keyword, it doesn’t return 0 but noting. Something we had to take into account in our logic.

That being said, let’s have a look at the code:

async function get_search_vol(chunk, url) {
  const requestUrl = url; // URL to request
  const response = await fetch(requestUrl); // Make request to Keyword Surfer
  const json = await response.json(); // Transform response to JSON
  //loop the response and return an array with volumes
  let keywords = {};
  for (i = 0; i < chunk.length; i++) {
    keywords[chunk[i]] = json[chunk[i]]?.search_volume ?? 0; // If keyword has data get the data else return 0 (optional chaining operator (?.) + Nullish coalescing operator (??))
  }
  return keywords;
}

//function to divide an array in X arrays
//Source: https://ourcodeworld.com/articles/read/278/how-to-split-an-array-into-chunks-of-the-same-size-easily-in-javascript
function chunkArray(myArray, chunk_size) {
  var index = 0;
  var arrayLength = myArray.length;
  var tempArray = [];

  for (index = 0; index < arrayLength; index += chunk_size) {
    var myChunk = myArray.slice(index, index + chunk_size);
    // Do something if you want with the group
    var finalChunk = [];
    for (y = 0; y < myChunk.length; y++) {
      finalChunk.push(myChunk[y].replace('"', '').replace('"', '').replace('&',''));
    }
    tempArray.push(finalChunk);
  }

  return tempArray;
}

//generate the Keyword Surfer's URLs for our chunks
function generate_urls(chunks, country) {
  //output
  const arr = [];

  //Base URL to generate our list of urls
  const base_url = `https://db2.keywordsur.fr/keyword_surfer_keywords?country=${country}&keywords=[%22`;

  // Loop through chunk to create array of keywords in request
  for (i = 0; i < chunks.length; i++) {
    var url = base_url.concat(chunks[i].join('%22,%22'), '%22]');
    arr.push(url);
  }
  return arr;
}

To extract volumes, there are several steps:

• A function called chunkArray to divide our list of keywords in chunks of X elements. We need to perform that action because we can request up to 50 keywords to Keyword Surfer’s API.
• A function called generate_urls to create a list of requests URLs using our chunks. The structure is pretty simple: https://db2.keywordsur.fr/keyword_surfer_keywords?country=us&keywords=[%22buy%20car%22,%22buy%20cars%22] (example with 2 keywords), even if special characters (see this reported issue in GitHub) can break the API and are currently the main reason why users can’t use the extension properly sometimes.
• A function called get_search_vol to retrieve the search volume. Indeed, Keyword Surfer’s API provides us with more than just the search volume, we therefore need to process the result to extract only the information we need.

We then add these functions altogether inside another one to obtain the expected output: a list of keywords (from GSC) with their associated volume (from Keyword Surfer’s API).

async function getData(country) {
  const kws = get_keywords(); // Get all keywords from GSC
  const chunks = chunkArray(kws, 50); // transform keyword in multiple arrays of 50 keywords
  const urls = generate_urls(chunks, country); // Build Request URL
  const allKeywords = {}; // Store future reponses in hashmap

  // Loop through GSC set of keywords and request keywoFrd surfer data
  for (let i = 0; i < urls.length; i++) {
    var sv = await get_search_vol(chunks[i], urls[i]);
    var keys = Object.keys(sv);
    for (let y = 0; y < keys.length; y++) {
      allKeywords[Object.keys(sv)[y]] = sv[Object.keys(sv)[y]];
    }
  }
  // console.log(allKeywords); // Just to check the output
  return allKeywords;
}

Amazing! The only this we now need is to display this information to the user!

Add them to the UI

function createCell(text) {
  var cell = document.createElement('td');
  var cellText = document.createTextNode(text);
  cell.appendChild(cellText);
  cell.setAttribute(
    'style',
    'font-size:12px;font-weight:bold;text-align:center;padding:18px 28px;'
  );
  return cell;
}

async function addVolumes(country) {
  const volumes = await getData(country); // Wait to get hasmap of search volumes
  const tbl = document.getElementsByClassName('i3WFpf')[0]; // Select table
  // Future CSV
  let csvExport = '';

  // Loop through rows
  for (let i in tbl.rows) {
    // For some reason there is an undefined row at the end
    if (i === 'length') {
      break;
    } else {
      // Select each row
      let row = tbl.rows[i];
      // Create line for future CSV
      const line = [];
      // Select first cell (query)
      const query = row.cells[0].textContent;
      // Add header
      if (query === tbl.rows[0].cells[0].textContent) {
        row.appendChild(createCell('Search Volumes'));
      } else {
        // If there is search volume data add it
        if (volumes[query]) row.appendChild(createCell(volumes[query]));
        // If not add 0 search volume
        else row.appendChild(createCell(0));
      }
      // Loop through cells
      for (const cell of row.cells) {
        const text = cell.textContent;
        line.push(text.replace(/,/g, ''));
      }
      // Create each filled line for future CSV
      csvExport = csvExport.concat(line.join(','), '\n');
    }
  }
  createDownloadButton(csvExport);
}

The logic is pretty simple here:

• A function called createCell is used to create cells in our GSC table. The CSS added (inline) in this function can be seen in your source code once the volumes have been retrieved by the extension.

• A function called addVolumes add the volume column to our entire table. If the keyword has no data in Keyword Surfer (as mentioned above), we add 0 by default.

What about user input?

If you used my extension, you remember that you need to specify your country to use it.

Storing an user input to be reused later on was actually the most difficult part of the whole process. The code may look simple but:

• The documention is not very clear and understanding why your code doesn’t work is not easy either, because it often doesn’t trigger any error
• Google Chrome has its own standards instead of using JavaScript’s

That being said, the following code allows my extension to save the user input, to be later on used by the main code to retrieve search volume for a given country. In my example, it would retrieve volume for France.

const optionsStatus = document.getElementById('status');
const countryList = document.getElementById('country');
const optionsSaveBtn = document.getElementById('save');

function save_options() {
  var country = document.getElementById('country').value;
  chrome.storage.local.set({ country: country }, function () {
    // Update status to let user know options were saved.
    var status = document.getElementById('status');
    status.textContent = 'Options saved.';
    setTimeout(function () {
      status.textContent = '';
    }, 750);
  });
}

// Restores select box and checkbox state using the preferences
// stored in chrome.storage.
function restore_options() {
  // Use default value color = 'red' and likesColor = true.
  chrome.storage.local.get(
    {
      country: 'fr',
    },
    function (items) {
      document.getElementById('country').value = items.country;
    }
  );
}
document.addEventListener('DOMContentLoaded', restore_options);
document.getElementById('save').addEventListener('click', save_options);

How to debug an extension?

When you create a JavaScript code to be used by an extension, you need to test & debut it first. You have two options:

• Use the code directly in Chrome: you can run directly JavaScript using the Console tab of Chrome DevTools. Pretty handy at the beginning (because you don’t have to install anything) but it forces you to copy-paste your code quite often.

• Install your extension: you can install an unverified extension (follow these steps) and use it as a regular user would. It requires you to build the manifest.json though, something you may not want to create at the beginning of the process.

Maybe there are most efficient way to debug an extension, but as a newbie I only used these two, and it was more than enough for my project.

How to submit your extension?

If you want to publish your extension, you can follow the official documentation which explains it pretty well. To summarize the process, you’ll have:

• To create a developer account and pay the 5$ fee (even if you extension is free)

• Submit your extension by providing the required information, like its purpose. Please note that you need to justify the permission required by your extension. I actually got my second submission rejected because I required too many permissions when it was not necessary. So be careful with that, because it will slow the full process.
• Don’t include the word “Google” in your extension name (or any other brand name). My first rejection was caused by this rookie mistake, Google considering that you pretend to be someone else (Google in my case) by using its name.

Apart from these recommandations, the validation process is rather quick (less than a week). If you want to update your extension, the process is even quicker as you obviously don’t need to fill everything again.

Conclusion

Improving my workflow is always one of my priority and despite coming from a Python background, developing this extension was not as hard as I thought. I wouldn’t have achieved it without the help & advice of the persons I mentioned at the beginning of the article though, let’s be honest here Extensions are easy to create and can save us a lot of time, so that’s definitely something I’ll be looking at if you know a bit of JavaScript.

Python is popular right now, but we can’t deny that JavaScript has its advantages.

The post Build a Chrome extension to automate a part of your routine appeared first on Antoine Eripret.

There are several tools to help us in this task, however we tend to overlook the most obvious one: Google Suggestions.

What is Google Suggest?

Everyone is using Google Suggest, most without even knowing it. When you start searching for something on Google, suggestions usually appear based on:

• The version of Google you use
• Your language
• Your search history

Although no tool will indicate an interesting search volume for this search, I know it will have plenty over the next few days. So does Google, which is why it suggests it to me.
It is interesting, but extracting data from this tool manually can be tedious. It’s true, however there is an API that allows us to speed up the process a bit. The best part? You can scrape without complex configuration.

TL;DR

If you are not interested in knowing how to perform this extraction and you are only interested in the result, you can access directly to the Google Colab I created and follow the instructions.

If you are interested in the details of the code (made in Python), you can continue reading.

Explanation of the code

The code is quite simple, but I am going to explain each of the steps so that you can understand it and adapt it if you wish.

Loading libraries

First, we have to load the different libraries. Some of them are quite well known (Pandas, Requests, JSON and BeautifulSoup) and you can find tutorials on the internet without any problem. The only not so common functionality I use is the ascii_lowercase function of the string library, which allows me to easily create a list with the different letters of the alphabet.

Why do I need it? You will understand this in one of the following sections.

#load libraries
import pandas as pd 
import requests
from bs4 import BeautifulSoup
from string import ascii_lowercase
import json 

Enter the seed keyword and country

The Google Suggest API works only if you indicate a word, so that it returns suggestions. Therefore, you need to indicate one. In the Google Colab (which I shared with you at the beginning of the article), you will see that you can, as a user and without modifying the code, indicate the word you want.

This is possible thanks to the #@param {type: “string”} that I include at the end of each of the lines. It tells Google Colab that the user can modify the value of this variable. You can read more about this functionality here.

#Get input from users
seed_kw = "como taladrar" #@param {type:"string"}
country = "es" #@param {type:"string"}

Extract suggestions

This part allows you to extract suggestions using the Google Suggest API.

What does it do exactly?

• It creates a list of keywords, adding the letters of the alphabet after your initial word. If you enter shoes, you will get shoes a, shoes b etc…
• Add to this same list, and following the same logic, your keyword followed by the numbers from 0 to 9.
• Extract the keywords returned by the Google Suggest API for each of these combinations.

Why do I follow this process? Simply because the Google Suggest API returns at most 10 suggestions. And if we add letters / number after your initial word, it allows to get more suggestions in a single run.

#convert seed keyword to list
keywords = [seed_kw]
#create aditionnal seeed by appending a-z & 0-9 to it
for c in ascii_lowercase:
  keywords.append(seed_kw+' '+c)
for i in range(0,10):
  keywords.append(seed_kw+' '+str(i))

#gett all suggestions from Google
sugg_all = []
i=1

for kw in keywords:
  r = requests.get('http://suggestqueries.google.com/complete/search?output=toolbar&hl={}&q={}'.format(country,kw))
  soup = BeautifulSoup(r.content, 'html.parser')
  sugg = [sugg['data'] for sugg in soup.find_all('suggestion')]
  sugg_all.extend(sugg)

Extracting search volumes

At the beginning of this article, I mentioned that search volumes are not always useful. It is true, but sometimes it can be interesting to have this information. In the end, you can also extract data from Google Suggest to develop the architecture of an E-commerce.

There are many APIs that allow you to extract this data, but for this tutorial I have decided to use the one proposed by Keyword Surfer:

• It is free
• It does not require a password (create an account)
• It does not have many limitations
• It allows to extract volumes for up to 50 words in a single request

Now, it’s not perfect, especially for words with a search volume lower than 50 monthly searches, but you can replace it with another API if you want. This part of the code extracts the search volume for each of the suggestions returned by the Google Suggest API, for the country you indicated in step 2.

#remove duplicated
sugg_all = pd.Series(sugg_all).drop_duplicates(keep='first')
if len(sugg_all)==0:
  print('There are no suggestion. The script can\'t work :(')
  
#get search volume data 
#prepare keywords for encoding
data = sugg_all.str.replace(' ','%20').unique()
#divide kws into chunks of kws
chunks = [data[x:x+25] for x in range(0, len(data), 25)]

#create dataframe to receive data from API
results = pd.DataFrame(columns=['keyword','volume'])

#get data 
for chunk in chunks:
  url = (
      'https://db2.keywordsur.fr/keyword_surfer_keywords?country={}&keywords=[%22'.format(country)+
      '%22,%22'.join(chunk)+
      '%22]'
  )

  r = requests.get(url)
  data = json.loads(r.text)

  for key in data.keys():
    results.loc[len(results)] = [key,data[key]['search_volume']]

Save the results

We are getting to the end! We only need to indicate when a word has no searches (by default, the Surfer API does not include words with a volume of 0) and save the result in a CSV that we can use later in Google Sheets or Excel.

Easy and fast!

The post Scrape Google Suggest at scale appeared first on Antoine Eripret.

This is true for a lot of industries but the one that has been impacted the most is the coupon industry. Here is a random query I ran, and out of the three first results, two actually belong to well-established news outlets.

In this article, we are going to explain why we see such a situation in a lot of markets. You have exactly the same situation in France, Italy, Germany …

Newspapers in a digital-first world

Before tackling the main subject of this article, we must explain why we often see these subdomains or subfolders linked to newspapers and no other companies.

Some years ago, this industry was running on an advertising-only business model. With radio and TV, it was one of the most widespread ways of marketing a product or a brand. This is actually why we saw, in the early 2000s, the birth of free newspapers: the advertising money was enough to break-even: you didn’t have to charge for the content you were producing.

Today, the reality is very different: the market share of printed advertising has fallen dramatically. You can see the data for Spain below, extracted from an amazing presentation on SEO for newspapers (in Spanish) that you should definitely check out if you are interested in this matter and can speak the language.

And let me highlight what you already know: the internet advertising budget is not allocated anymore to newspapers: the vast majority is used on Google, Facebook, and Amazon. They still get a piece of the cake, but a (way) smaller part.

At the same time, newspapers had to create and maintain a digital presence, which obviously increased their costs. The equation was not working anymore and they had to find alternative ways of earning money. The most famous one was the creation of subscription plans, where you would pay a monthly or annual fee to access exclusive content, which came with its own challenges:

• To compensate for the loss of advertising revenue, you need a lot of premium subscribers, an objective that a lot of newspapers fail to achieve. The only success story I can think of is the New York Times (see full story)
• A lot of internet users refuse to pay to consume content for two reasons: they don’t want to (they believe everything on the internet should be free) or because they sometimes don’t see the added value versus free content (which to be honest is sometimes true).

Once newspapers realized that they weren’t going to break-even with only a subscription-based business model, they started to look for alternative solutions. One of them was to leverage their popularity to offer a subdomain or subfolder leasing to a third-party.

You don’t know what the heck is a subdomain leasing? Don’t worry, let’s just go through that.

What is a subdomain leasing?

When you own a domain, you can create a huge number of subdomains easily. A subdomain leasing is simply when one of these subdomains is leased to a third-party and is ran independently.

Coupon sites are one example: a newspaper will create the subdomain and lease it to a company for a fee. In some cases, both parties are involved more in a partnership than a regular lease.

That is great, but what is even the point of paying for a subdomain if you can create your own domain? Let me explain why by showing an example in the Spanish market.

If you compare the visibility of two of the biggest names in this industry, you can see a normal trend for any given industry: you have fluctuations caused by changes applied to the project or external factors (a new competitor for instance), but nothing really surprising.

Now if I add two leased subdomains, you can clearly see that they achieved to keep up and then overcome these two websites in a matter of months when the industry is really competitive.

Let’s not explain just now how they can rank so fast, but you can understand why a company would be willing to pay just to use a subdomain of one of these newspapers.

What is a business model behind a cupon website?

If you are not familiar with this industry, let me take a moment to explain how these websites earn a living. At the end of the day, if newspapers are interested in running o leasing a cupon subdomain (or subfolder), it is because they can make money.

There are several ways of making money, but the most common is through affiliation. When you redeem a coupon code as a user, two options:

• A cookie is added to your navigator to identify the cupon website as the affiliate when you make a purchase
• You are redirected to the website where you want to purchase something and the coupon website is identified thanks to one URL parameters

In both cases, the coupon website will get a commission out of the purchase you will make. What is funny is that the commission can sometimes be earned even if the coupon doesn’t work. Which explains why so many coupons are expired or not valid.

Given the traffic you can achieve with these websites, it is very clear that you can make some real money in this industry

How do leased subdomains (or folders) rank so fast?

Back in 2018, Google warned that leased subdomains could get hammered. In reality, little happened because these subdomains are not worse than others. From a user perspective, they deliver the same thing: a coupon code. You can’t assimilate it to spamming, it is different.

So, how do these websites rank so fast? If we look at the number of links reported in Ahrefs for two major players in Spain, we actually see that cuponation.es has more and more domains.

Yes, but the trick is not to look at how many but from where and how. I will use descuentos.abc.es to explain it.

Like in a lot of countries, you have a lot of newspapers in Spain but they actually belong to a reduced number of media companies. If you want an example for the US, just look at this amazing article written in 2016.

So, the group behind ABC (Vocento) also owns several other brands, including one called La Voz Digital.

This website has a link in its footer, using the keyword “Descuentos”, linking to… descuentos.abc.es. Not very surprising, I know.

On the main ABC website, you also have a link in the main navigation using the keyword “Cupones descuento”. Again, to descuentos.abc.es.

You got the point, don’t you? These websites are basically leveraging their network to rank new domains/subdomains fast. And of course, Google will rank faster a website receiving a decent number of links from these popular websites than a new brand. Links do matter and I explained it an older post.

For newspapers, being able to rank quickly a website on a new vertical is great and it is completely normal that they leverage their capacity to do so. I mean, you have countless of examples of other industries where the German domain would link to the Italian domain using the exact keyword in Italian for instance. It is exactly the same. So why would we blame newspapers for doing that?

It also solves a part of the problem we mentioned at the beginning of this article. They can launch new businesses and diversify their income stream. But let’s be honest: they still to launch a few to compensate the shrink of their physical advertising revenue. They also need to be careful: you can’t ruin your brand reputation by leasing your subdomain to a bad third-party. That’s exactly why they actually don’t launch a dozen of new verticals at once, that would be quite risky.

For competitors, it means that you need to provide something else than just the cupon to compete with these websites. As you can’t compete with Amazon if you offer exactly the same, you need to differentiate yourself and work hard on your branding. Something that achieved the brand Chollometro in Spain for instance: they don’t just offer coupon, they also provide huge bargains. And everything is validated by the community to ensure it is real !

Is it the only thing happening with their subdomains?

Obviously, no. In terms of SEO, the news industry is very different, because you basically do a lot of real-time SEO. The vast majority of your content will generate traffic for less than 72H. Yes, you can create hub pages on key topics (coronavirus, world cup, etc…), but this doesn’t account for the majority of their traffic.

Hence, the news industry started to acquire (I don’t think these cases are leased but I may be wrong) existing projects and include them as subdomains, to rank on evergreen keywords as well.

Why? You don’t have to create everything from scratch and you can’t ensure that a part of your traffic comes from keywords with more stable search volume over time. Again, the idea is to diversify their stream of income because the old paradigm is broken and cannot be applied to the digital world.

You can also create new business models around these subdomains. If you have a recipe section, you can create a partnership with an online supermarket to allow your users to order the ingredients from them for instance.

Conclusion

Subdomain leasing is here to stay, so we better get used to it. I don’t think it is a bad thing per se, because again these websites are fulfilling the search intent. They don’t rely only on the backlinks to rank, even if it helps a lot.

Is it unfair? Big brands are always advantaged: Amazon would rank easily on a new commercial term than any other retailer for instance, and I don’t think this situation is any different. If you have a different opinion, I would happy to discuss it with you in the comment section though

The post How newspapers leverage subdomain leasing? appeared first on Antoine Eripret.