Technical SEO Basics: Speed, Mobile, Crawling and Security

Key takeaways

• Technical SEO makes sure search engines can reach, understand and trust your site, and that visitors get a quick, secure experience.
• robots.txt manages crawling and is not a way to keep a page out of Google; use noindex for that, and do not block the page in robots.txt.
• Google uses the mobile version of content for indexing, so keep the same important content and links on mobile as on desktop.
• Good Core Web Vitals are LCP within 2.5 seconds, INP of 200 milliseconds or less and CLS of 0.1 or less, at the 75th percentile.
• Use permanent redirects when a URL moves, name a canonical URL for duplicates and review the indexing report on a schedule.

What technical SEO is

Technical SEO is the work of making sure a website can be found, read and used smoothly by search engines and visitors. It is not about writing content. It is about the foundation underneath the content: how pages are served, how quickly they appear, how they behave on phones and whether search engines can reach them without obstacles.

You can have excellent articles and still struggle if the site blocks crawlers, loads very slowly or shows errors. Think of technical SEO as the building's wiring and plumbing. Visitors rarely notice it when it works, but they certainly notice when it fails.

How do you make a site crawlable?

Search engines must be able to reach your pages. Several simple files and settings control this.

• robots.txt: a text file at the root of your site that gives crawlers instructions about which areas to avoid. It is helpful for admin areas or internal search pages, but one wrong line can block your whole site, so review it with care.
• XML sitemap: a list of the important URLs you want search engines to know. Keep it clean: include only pages you want indexed and that return a normal response.
• Internal links: a crawler follows links. Pages with no links pointing to them, sometimes called orphan pages, are hard to discover.
• Site structure: a logical hierarchy where important pages are only a few clicks from the homepage helps both users and crawlers.

Which controls decide what gets indexed?

You can tell search engines how to treat a page.

• Noindex: an instruction that asks the search engine not to store a page in its index. Useful for thank-you pages or private results, but dangerous if added by mistake.
• Canonical tag: a line of code that names the preferred URL when similar pages exist. It helps when the same content can be reached with different parameters or paths.
• Redirects: a 301 redirect tells browsers and search engines that a page has moved permanently. Use it when you change a URL.
• Status codes: every request returns a code. A 200 means success, a 404 means not found, and a 5xx code means a server problem. Pages that are removed for good should return a sensible response rather than a misleading success.

Why does HTTPS matter?

HTTPS is the secure version of HTTP, the protocol that browsers and servers use to exchange pages. The S stands for Secure. It encrypts the connection between the visitor and your website using a certificate. Browsers warn people when a site is not secure, which can reduce trust, and search engines have long encouraged HTTPS. Install a valid certificate, make sure all pages and resources load over HTTPS, redirect the HTTP versions to HTTPS and keep the certificate renewed. Keep your website software updated as well, since hacked sites can lose visitors and visibility.

Why does mobile friendliness matter?

Many Indian internet users browse on smartphones, and search engines primarily evaluate the mobile version of pages. A mobile-friendly site has these qualities.

• A responsive layout that adapts to different screen sizes.
• Readable text without pinching and zooming.
• Buttons and links that are easy to tap and not too close together.
• The same important content and links on mobile as on desktop.
• No intrusive pop-ups that cover the content.

Test your pages on a real phone, preferably on a mid-range device and a modest connection, since that is closer to what many visitors experience.

What are Core Web Vitals?

Speed shapes user experience. Core Web Vitals, abbreviated CWV, are a set of measurements Google uses to describe real-world page experience. They look at three ideas.

• Loading: how quickly the main content becomes visible, measured by Largest Contentful Paint, or LCP.
• Interactivity: how quickly the page responds when a user taps or clicks, measured by Interaction to Next Paint, or INP.
• Visual stability: whether elements jump around while loading, measured by Cumulative Layout Shift, or CLS.

Google publishes target thresholds for these metrics, and you can see how your pages perform in the Core Web Vitals report in Search Console and in tools such as PageSpeed Insights. Treat them as indicators of a smooth experience rather than as a guarantee of any ranking position.

Core Web Vitals at a glance

These thresholds are assessed at the 75th percentile of page loads, separately for mobile and desktop.

Common ways to improve speed

• Compress and properly size images, and consider modern formats.
• Load only the scripts and styles a page needs, and delay the rest.
• Use browser caching so repeat visitors do not download everything again.
• Use a content delivery network, or CDN, which stores copies of your files on servers in multiple locations so they reach visitors faster.
• Choose reliable hosting with servers reasonably close to your audience.
• Reserve space for images and ads so the layout does not shift.

How does JavaScript affect crawling?

Many modern sites build their content using JavaScript. Search engines can process JavaScript, but it adds a step and can fail if scripts break or content depends on user actions. Where possible, make sure the key content and links appear in the page's HTML when it loads, or confirm with a URL inspection tool that the search engine sees what visitors see.

Structured data

Structured data labels the meaning of page elements in a standard vocabulary. It can describe articles, products, organisations, events and more. Use it accurately, test it with the available validation tools and only mark up information that is visible on the page.

How do you handle duplicate and thin pages?

Websites, especially shops with filters and sorting options, can generate huge numbers of near-duplicate URLs. Decide which pages deserve to be indexed. Use canonical tags, sensible internal linking and, where appropriate, noindex to guide search engines towards the valuable versions. Remove or improve pages that provide little value.

What about international and multilingual sites?

If you publish in more than one language or for more than one country, tell search engines about the relationship using hreflang annotations, and keep the structure of language versions clear. Many Indian businesses serve audiences in English alongside Hindi or other regional languages, so this can be relevant.

What does a simple technical audit look like?

• Open the indexing report in Search Console and review excluded pages and errors.
• Check that your sitemap is submitted and contains the right URLs.
• Run a crawl with a free or paid crawler to find broken links, redirect chains and duplicate titles.
• Test a few key pages on mobile and review speed results.
• Confirm that HTTPS works everywhere.
• Repeat on a schedule, and after any major site change.

In short

Technical SEO ensures that search engines can reach, understand and trust your site, and that visitors have a quick, secure and comfortable experience on any device. Focus on crawlability, correct indexing controls, HTTPS, mobile usability and reasonable speed. Fix the biggest problems first, check regularly with free tools, and remember that a sound technical base lets your content do its job.

Sources and further reading

• Google Search Central: Introduction to robots.txt: states that robots.txt tells crawlers which URLs they can access and is not a mechanism for keeping a page out of Google.
• Google Search Central: Block Search indexing with noindex: explains the robots meta tag and X-Robots-Tag header and that the page must not be blocked by robots.txt for noindex to work.
• Google Search Central: Consolidate duplicate URLs: describes rel=canonical and permanent redirects as strong signals and a sitemap as a weak signal of the preferred URL.
• Google Search Central: Redirects and Google Search: explains that permanent redirects signal the target as canonical while temporary ones do not, and that server-side redirects are preferred.
• web.dev: Web Vitals: gives the good thresholds LCP within 2.5 seconds, INP of 200 milliseconds or less and CLS of 0.1, assessed at the 75th percentile of page loads.
• Google Search Central: Understanding Core Web Vitals and Google Search results: recommends good Core Web Vitals for success with Search and points to the Core Web Vitals report in Search Console.
• Google Search Central: Mobile-first indexing best practices: states that Google uses the mobile version of content for indexing and ranking and advises equivalent content on mobile and desktop.
• Google Search Central: JavaScript SEO basics: explains how Google crawls, renders and indexes JavaScript pages and that links need an href attribute.
• Google Search Central: Localized versions of your pages: explains hreflang, the need for return links and the x-default value.

Frequently asked questions