It can often be difficult to get an overview of which SEO measures can be improved on your website. In fact, it can be overwhelming to think about where to start with the technical and visual analysis of your site.
In this guide, you'll get an introduction and of course tips for the SEO application loved and appreciated by all SEO specialists, generalists and digital marketing consultants worldwide - namely Screaming Frog. Once you've read this guide, you'll be ready to improve your website - and have a positive impact on search engine optimisation. Welcome to!
Screaming Frog is an application that can troubleshoot and visualize technical issues on your website. It works like a spider, similar in some ways to the way a search engine such as Google crawls your website. It's clever because it allows you to improve challenges on your website - and optimise them in a positive direction.
The program is available for PC (Windows), Mac and Linux, and is available in a free and paid version. In the free version, you can crawl a maximum of 500 URLs, and various technical features will not be available - for example, you cannot see which pages have structured data implemented or integrate the data from Google Analytics.
If you're not familiar with structured data, I've written an introduction to the topic right here.
Anyway, back to Screaming Frog! The premium version of the program costs £149 (about 1235 dkk) per year, which gives you access to all features. However, you might want to test the free version for a while and then decide whether a licence is necessary. However, it is again worth noting that you only have limited access to various features in the free version.
Screaming Frog can do a lot of things. Some of the features are difficult to use, while others are significantly easier to get to grips with. I've collected seven tips that are mainly easy to use. Of course, it depends on which CMS you use. For example, if you use WordPress, then you're already in a good position to improve your website SEO-wise.
1. Check if metatags are the correct length - and catchy!
2. Get an overview of your image sizes and alt-texts
3. Find broken links (404 pages)
4. Analyse redirects when migrating a page
5. Integration of G Suite products and other SEO tools
6. Crawl your page the way Google wants to
7. Verification of your schema markup
It's important that your meta tags are the right length, but also catchy. When your website is exposed on Google based on a specific search criterion, you only have a certain number of characters to make good use of. An example is the following:
The above example shows that the meta description is too long and needs to be corrected. But what if you have several pages where the meta description or title is too long? To find out, this is where we turn to Screaming Frog. By entering the domain into the program and then navigating to Overview -> Meta Descriptions we get an insight into which meta descriptions are too long, too short, duplicated or other critical things that need to be improved.
Once you have an overview of which pages need to be optimised, you can find them in the backend of your website. If you use WordPress, you can install Yoast SEO (I wonder if you've already done that? ;)), and navigate to SEO -> Tools -> Mass-editing. Here you can edit your meta titles and descriptions in bulk - which is easier than clicking into each page.
At Webamp , we've even created a small application to help you write compelling meta-titles and descriptions. This can help increase your CTR (click-through-rate) and thus increase the number of visitors via the organic search results on Google. You can find our CTR tool right here.
A website needs to load quickly. That's why there are demands on how your content loads - especially images. The most popular search engine, Google, primarily analyses your website via the mobile version - also called mobile-first indexing. So what does this mean for you? It means you need to focus on how images load, how much space they take up and what format your images are in.
In this context, alt-texts are also important. In short, alt-texts describe your images in a written context. This is particularly relevant for users with partial or full sight impairment, or users who use a screen reader or have a poor internet connection. This allows them to get a description of what the image contains and means if they cannot see it.
You can read much more about what Google recommends as best practice in image optimisation right here.
Posting a picture on a website is therefore not as simple as uploading a party picture on your favourite social media. You need to take into account the size of the image itself, as well as its dimensions. Via Screaming Frog, you can see both how much your images take up and which ones are missing alt captions. This can be found under Overview -> images.
Once you have found the list of images, you can sort them by size and then click on ''Inlinks'' at the bottom of the application. This will give you an idea of which images are taking up the most space and where they are located on your website.
With Screaming Frog, you also have the option to find broken links that point to either external sources or previous subpages on your site. It is generally accepted that these links are fixed so that they are redirected to the correct new page or equivalent content that appeared on the now ''broken'' page. Not only for the sake of a search engine, but also for the user experience.
For example, it is very obvious to shut down the site if you are not redirected to the right content - this will create a poor user experience all other things being equal. That's why it's important to get these links right. This is how you do it:
As with the list of images, you have the option to find the location of the links on your site - and in any anchor text.
In addition to 404 pages, you can also see which redirects actually work, which links are blocked by robots.txt and much more. In general, you should be aware of what response codes your server sends to the client visiting your website, as this can influence how a search engine indexes it.
If you've created a new website and want to ensure that the existing SEO value will be ''transferred'', it's important that your redirects are just right. Otherwise, you could potentially lose SEO value if Google indexes one of your old landing pages high and you don't tell the search engine that the content may have changed location.
With Screaming Frog you have the possibility to make sure that the migration from the new to the old site is done in a proper way and that all redirects are done.
First you need to switch modes from Spider to Cunning. This is done under Mode in the menu:
Once this is done, upload the list of your old URLs. There are several options, but I have chosen to do it manually:
Then you need to set the program to follow your redirects. This is done under Spider -> Advanced -> Always Follow Redirects.
Once the crawl has been completed, select Reports -> Redirects -> All Redirects, after which you can download a .CSV file, which you can import into e.g. Excel or Google Sheets and analyse the results there.
For the sake of clarity, I've made a sample where you can see that one of Webamp's clients' redirects is done properly and sends traffic to the desired URL.
Through Screaming Frog, you actually have the possibility to combine several different kinds of data and thus analyse how to optimise your website. An example could be the following:
Landing pages with missing meta tags (see Screaming Frog) often have a low CTR (click through rate), as the user is not influenced to visit the pages. By also looking at the amount of traffic and the average ranking of the specific page, you can benefit from optimising for this. In this case, you use Screaming Frog to find pages with missing meta tags, while interacting with the data from Search Console (average rankings and clicks) and Analytics (traffic to the page) respectively.
Via Screaming Frog you have the possibility to connect to the following APIs:
To connect to various APIs, before you start crawling your website, you need to connect to them. This is done via the API function on the right side:
In this example I have done it with three different ones; Google Analytics, Google Search Console and Ahrefs. Next, I have the ability to compare data from the three sources to analyze each landing page and how they can be optimized.
Screaming Frog has its way of crawling a website when it needs to visualise and tell you how to optimise it. But in fact, you can also have the program crawl the website as if it were Google. Why is that relevant, you ask?
That's because it's Google that decides how your website will be indexed - and therefore how it will rank. Furthermore, the search engine also indexes your website based on a mobile-first principle (as mentioned earlier in this blog) - i.e. it crawls your website as if it were a mobile device. That you can change this setting in Screaming Frog is important because of two factors:
1. By mimicking the Googlebot Smartphone crawler, you can quickly find out what problems Google may have when crawling and rendering your website from a mobile-first principle
2. Using the Google Smartphone User-Agent in the application, you can compare your crawls and Google's own when analysing server logs - to see where the difference lies.
This is done relatively easily. By choosing Configuration -> User-Agent in the menu, you can easily switch this.
Structured data is the new black, if you ask many SEO specialists. This should be seen in the light of the fact that it can pave the way for your website to become more visible in organic search results. With structured data, you can do the following:
However, this data needs to be validated so that it is set up correctly - and you can use Screaming Frog to do this. Before you start crawling the website, navigate to Spider -> Extraction in the menu. Here you can select JSON-LD, Microdata, and RDFa respectively. However, it depends on how you have chosen to structure your data. For the sake of order, I have chosen them all. You can also choose Schema.org Validation and Google Rich Result Feature Validation, as you don't have to go to Google's own validation tool to see if the data you have structured will appear on the search engine.
After the program has done its work, you can take a look under Overview -> Structured Data to see if there is anything that needs to be fixed:
Those were the seven tips I had to offer. You can use Screaming Frog for much, much more, but these tips will at least equip you to optimise the most basic of opportunities - especially if you're only now starting to explore search engine optimisation and your options for increasing organic traffic.
By researching which keywords your main competitors are targeting, you can get inspiration for your own keyword analysis.
Do they rank on keywords you should rank on too? And are there gaps in the market?
Whether you're a generalist or a marketing specialist, our SEO specialists have put together some great advice for you on our blog.
Learn more about SEO in Webamp Academy.