Wednesday, December 21, 2016

Do I need to build a Sitemap for my Website?

What is a sitemap?

Human readable sitemaps are static HTML files that outline the first and second level structure of a Web site. The original purpose of a site map was to enable users to easily find items on the Web site.  But the purpose grew over time to became useful as a shortcut method to help search engines find and index all the parts of a site.

Now, we have an XML sitemap, which effectively provides an easy-to-read link dump for the spiders to index. Some Web browsers can display an XML sitemap for users to read as well, you should offer both kinds of site maps (HTML and XML) if you want to be sure to cover both the search engines and your users.


What is a Sitemap used for?

Building a website can be a long tedious process, which can be more complicated by the huge amount of information that should be organized before including in your website. Some designers will start by creating  wireframes and mockups.  But for the rest of us it a lot easier to just build a sitemap to use as a guide to build the structure and pages needed for your website.  The key factor is to keep you organized and to keep you from duplicating pages.

So What are all the types of sitemaps?

  1. standard HTML sitemap - Human readable list of your website links
  2. Text Sitemaps - Text sitemaps contain one website url per line. Many search engines including Google and Yahoo can scan text sitemaps. 
  3. Standard XML sitemap - Started by Google in 2005. The protocol of XML sitemaps also defines autodiscovery. It allows search engines can automatically discover website via the XML sitemaps.
  4. Image sitemap - Use to build a lost of images for the search engine like Google.
  5. Video sitemap - used to build list of your video for search engine like Google.
  6. Mobile sitemap - used to build list of mobile only pages for search engine like Google.  Useful for AMP , WAP or cHTML sites.
  7. News sitemap - This is used by Google News and is a great way to index your files for Google News.
  8. RSS Feeds as Sitemaps - RSS protocol is often used in feed files for blogs and forums. The RSS file format uses XML and has evolved over multiple versions and names, all fairly compatible with each other.
  9. ROR Sitemaps - ROR expands on the RSS protocol with its own extensions. Not currently supported by major search engine.

So Which sitemaps do I really need?

Basically there are only two sitemaps that you need to build.  Any of the other formats are optional & mostly used by bigger websites.
You only need 
  1. sitemap.html for our guests.
  2. sitemap.xml that you should submit to Google and the other search engines.  
You can also add this line below to your robots.txt file too.
Sitemap: http://www.your-domain.com/sitemap.xml


Here are examples of each sitemap formats:

Standard HTML sitemap  (sitemap.html)

<html lang="en">
  <head>This is a site map</head>
  <body>
    <h1>Your Site Name's HTML sitemap</h1>
    <p>Listed below are our the links for our website.</p>
    <a href="index.html" title="Title of your page">Home page</a>
  </body>
</html>
Text sitemap  (sitemap.txt)
http://www.your-domain.com/
http://www.your-domain.com/images/
http://www.your-domain.com/cgi-bin/


Standard XML sitemap (sitemap.xml)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc></loc>
    <priority>1.0</priority>
    <changefreq>weekly</changefreq>
    <lastmod>2007-06-18</lastmod>
  </url>
  <url>
    <loc>blogs/</loc>
    <priority>0.8</priority>
    <changefreq>weekly</changefreq>
    <lastmod>2007-06-21</lastmod>
  </url>
</urlset>


Image sitemap (sitemap-image.xml)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
  <url>
    <loc>http://www.your-domain.com/</loc>
    <priority>1.0</priority>
    <image:image>
      <image:loc>http://www.your-domain.com/1.png</image:loc>
      <image:title>image 1 title</image:title>
      <image:caption>image 1 caption</image:caption>
    </image:image>
    <image:image>
      <image:loc>http://www.your-domain.com/2.png</image:loc>
      <image:title>image 2 title</image:title>
      <image:caption>image 2 caption</image:caption>
    </image:image>
  </url>
</urlset>

Video sitemap  (sitemap-video.xml)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
  <url>
    <loc>http://www.microsystools.com/products/sitemap-generator/videos/sitemap-builder-sitemaps-tool/</loc>
    <video:video>
      <video:player allow_embed="yes">http://www.youtube.com/v/xZ-LrxSA3Yo?fs=1</video:player>
      <video:thumbnail_loc>http://i1.ytimg.com/vi/xZ-LrxSA3Yo/default.jpg</video:thumbnail_loc>
      <video:title>Sitemap Generator - Videos [ sitemap builder sitemaps tool ]</video:title>
      <video:description>Video: Sitemap Generator - build sitemaps quick demonstration</video:description>
      <video:category>sitemaps</video:category>
      <video:tag>a1</video:tag>
      <video:tag>sitemap</video:tag>
      <video:tag>generator</video:tag>
      <video:tag>google</video:tag>
      <video:tag>xml</video:tag>
      <video:tag>html</video:tag>
      <video:tag>sitemaps</video:tag>
    </video:video>
    <video:video>
    ...
    </video:video>
  </url>
  <url>
  ...
  </url>
</urlset>

Mobile sitemap (sitemap-mobile.xml)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0">
  <url>
    <loc>http://example.mobi/</loc>
    <priority>1.0</priority>
    <mobile:mobile/>
  </url>
</urlset>

News sitemap (sitemap-news.xml)

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:n="http://www.google.com/schemas/sitemap-news/0.9">
  <url>
    <loc>http://your-domain.com/</loc>
    <n:news>
      <n:publication>
        <n:name>News paper, blog or similar name</n:name>
        <n:language>en</n:language>
      </n:publication>
      <n:publication_date>2008-12-30T00:00:00+00:00</n:publication_date>
      <n:keywords>2008,news,xml,sitemap,example,keywords,generated</n:keywords>
      <n:title>News Article Title</n:title>
    </n:news>
  </url>
</urlset>


RSS Feeds as Sitemaps  (sitemap.rss)

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
  <channel>
    <title>Website title</title>
    <link>http://www.your-domain.com</link>
    <generator>Sitemap Generator</generator>
    <lastBuildDate>Tue, 13 Mar 2007 22:28:20 GMT</lastBuildDate>
    <item>
      <title>Page 1</title>
      <link>http://www.your-domain.com/page1.html</link>
    </item>
    <item>
      <title>Page 2</title>
      <link>http://www.your-domain.com/page2.html</link>
    </item>
  </channel>
</rss>

ROR Sitemaps (sitemap.ror)

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:ror="http://rorweb.com/0.1/">
  <channel>
    <title>Website title</title>
    <link>http://www.your-domain.com</link>
    <generator>A1 Sitemap Generator</generator>
    <lastBuildDate>Thu, 21 Dec 2016 08:01:30 GMT</lastBuildDate>
    <item>
      <title>Page 1</title>
      <link>http://www.your-domain.com/page1.html</link>
      <ror:keywords>page1-keyword1, page1-keyword2, page1-keyword3</ror:keywords>
      <ror:updatePeriod>day</ror:updatePeriod>
    </item>
    <item>
      <title>Page 2</title>
      <link>http://www.your-domain.com/page2.html</link>
      <ror:keywords>page2-keyword1, page2-keyword2, page2-keyword3</ror:keywords>
      <ror:updatePeriod>day</ror:updatePeriod>
    </item>
  </channel>
</rss>


The Sitemap Wrap up

Finally a sitemap can save you a lot of time by not creating unnecessary pages, or building sites that are overly complicated than needed.  Take time to sit down and create your websites content and figure out what is necessary.  Planning is key to saving time.  

You can find a number of great company on Google or other search engine.



Monday, December 12, 2016

Do I need a robots.txt File?

The robots.txt file

The robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.  

Here why it gets used; a robot wants to visits a Web site URL, say http://www.myorg.com/index/.  But, before it does so, it firsts checks for http://www.myorg.com/robots.txt, and find that it contains the following:

User-agent: googleboot
Allow: /
Allow: /images/
Allow: /css/
Allow: /js/
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /processors/
Disallow: /skins/

User-agent: bingbot
Allow: /
Allow: /images/
Allow: /css/
Allow: /js/
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /processors/
Disallow: /skins/

User-agent: *
Allow: /
Allow: /images/
Allow: /css/
Allow: /js/
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /includes/
Disallow: /processors/
Disallow: /skins/

sitemap: http://www.myorg.com/sitemap.xml

At MyOrg we break the file up to multiple sections


  1. For the GoogleBot - This allow us to give specific instruction to Google Bot Only.
  2. For the GoogleBot - This allow us to give specific instruction to Bing Bot Only.
  3. For all other Bots - This to gibe all other search bot instructions.
  4. Finally we include our Sitemap.  Using the sitemap directive you can tell search engines – specifically Bing, Yandex and Google – the location of your XML sitemap.


Explanation for why at MyOrg we setup our robots.txt file

The reason we would not just use the generic bot is because we may want to have Google or Bing index something that we don;t want othe sites to index.  This allows you great control of what get spidered.  And of course it always best to include the site,map so they know to index ever link you have in your sitemap.

To better understand why you use certain commands in the robots.txt file "vary.com" has already outlined a full explanation on robots.txt files I've ever read follow this link to vary.com's article titled "The robots,txt File"

A brief thought on why use proper site structure

When you are setting up your website if you create a structured site, then make proper use of the structure it will make your life easier.    Not only is it easier tot find things, but it also easier for the search engines to index your files,.  So instead pf just dumping all your files in the route directory or one sub folder; employ a ptroper site structure.

We at MyOrg find it easier to build folders for your Cascading Style Sheets (CSS), Javascript (JS)
Images, then add any additional folder you may need.  Examples of additional directries might be includes or processors or templates / skins.

I hope fiind this article  useful and give you a little insight on why you need a robots.txt file

Saturday, December 10, 2016

Could My Domain Be Blacklisted?

What is Blacklisting

In Internet terminology, a generic name for a list of e-mail addresses or IP addresses that are originating with known spammers. Individuals and enterprises can use blacklists to filter out unwanted e-mails, as most e-mail applications today have filtering capabilities.

Network administrators and users alike employ blacklists to block entities who would be likely to cause problems. The problem entities could be malware networks, spammers, hackers, DoS (denial of service) attackers or abusive site or forum users, among a plethora of other possibilities. Application blacklisting prevents the execution of undesirable programs, which include applications that are known to contain security threats or vulnerabilities and those that are deemed inappropriate for a given business environment. Hackers and IT security researchers sometimes use blacklists differently, seeking interaction with blacklisted entities to provide information.

How does Blacklist Works

 To help you understand how a blacklist works. If the incoming mail fails to satisfy criteria for legitimate mail, the IP can be included on blacklist. Some ISPs opt to use this data to inform their filtering decisions and delivery of the email messages sent over that Internet Protocol (IP).


How to Find Out if You’re on a Blacklist?

Having issues with your email or getting people tell you they can't get to your website and you know it is available? You could be blacklisted.   Have you been sending out a lot of mail, especially if you send bulk emails where there more than 25 recipients can cause you to get spammed by a receiver’s mail client or security software.  Or sometime when you use a hosting company the previous company that had the IP your domain was assigned got added to a blacklist.

I suggest you check your domain or IP with one or all of these companies.  But before you start know your IP



What is my IP & how do I Find my Domain's IP?

Every domain get assigned to an IP, most people don’t remember numbers like to do names.  So, domain name was introduced as a human-friendly equivalent of a IP address.   Most of the spam blockers or blacklist use IP's when the store a item to their list.  Here are a couple of sites you can look up your IP:

Offers "WHOIS IP Lookup Tool".  he IPWHOIS Lookup tool displays as much information as possible for a given IP address, sourced from the Regional Internet Registry (RIR) to which the address belongs.

This is one of the most versatile tools to find out the owner and internet provider of any website, domain or IP address. Locating IP addresses is useful for locating the origin of unwanted emails or the source of virus and port attacks.

 How to Get Off a Blacklist

High spam complaint rates are one of the main reasons that your IP ends up on a blacklist. So, it’s imperative that once you discover that you’re on one, that you take steps to get your IP removed from the list. By visiting each of these DNS-based Blackhole List  (DNSBL) sites, you can enter your IP to check your list status, and then obtain instructions on how to be removed. The key here is to be proactive &and work with each list you may be on.

How to Stay Off a Blacklist

Honestly the best practices is to help ensure that you only send wanted email and stay off of blacklists all together.  Your reputation is the key to proper email deliverability. By closely monitoring your complaint rates, you can prevent delivery failures before they happen. Check your stats with each campaign deployment. Test, test & test all new email campaigns.

I suggest you use email verification every time you update your list. Running email verification isn’t something that you should do only in a blacklist crisis, but a step you should take every time you send an email campaign.   Maybe use a company like Kickbox, Email verification service.  https://kickbox.io/


Thursday, December 1, 2016

Do You know the Robots META tag

Proper Robot Tag Usage

There is a special HTML meta tag to tell robots not to index the content of a page, and/or not scan it for links to follow.

Example Usage: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

There are two important considerations when using the robots Meta Tag.
  1. robots can ignore your Meta Tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  2. the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (maybe on some other site), and so still arrives at your undesired page.

How to write a Robots Meta Tag

Where to put it

Like any Meta Tag it should be placed in the HEAD section of an HTML page. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.

What to put into it

The "NAME" attribute must be "ROBOTS".  Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW".  Multiple comma-separated values are allowed, but only some combinations should be used.  If there is no robots Meta Tag, the default assumed value is "INDEX,FOLLOW", so there's no need to spell that out.  So That leaves the following values:

  • NOINDEX, FOLLOW
  • INDEX, NOFOLLOW
  • NOINDEX, NOFOLLOW
So double check what your hoping to accomplish and add your robot Meta Tag.

The Two Major Types of SEO

There are Two Major Types of Search engine optimization (SEO.)

On page SEO:

On Page is the Primary form of SEO.   This include making sure your website uses all the right HTML tags, structured data to correctly markup your content and is correctly formed or verified. This includes everything the words in the content you use and the code that forms the page. Think of it as: the key points of your page they are known as keywords. You have to use those keyword exactly in your text in order for people to find them. There also something called local SEO, where you know how to identify your business. And lastly the type of design you use, is it responsive? Yes there are even many more factors that need to be looked at, so hiring a professional can save you a lot of time. 

Off page SEO:

OK, so what does it mean Off the Page SEO? Off the Page is basically about building proper links to your site. You do not have a lot of control, but you have some ability to influence it. You can just pay someone to create backlinks, but in the long run they can hurt you. One method could be, posting articles in another quality site blog & providing a link to your site can help. Or get a similar type business to link to your site. 

Just be careful many business get penalized by Google for poor link building practices. 

It always best to hire a professional to at least double check you work.

Death of the Meta Tag Keywords

What Exactly Are Meta Keywords?

Keyword meta tags you could use in the HTML code of your site. In theory, you would use this tag to provide the search engines with more information regarding what your page is about. The search engines would then read the keywords in the tag and if the keyword is not in a target or header tahe it would think your spamming the search engine.

Keyword meta tags quickly became an area where someone could stuff often-irrelevant keywords without typical visitors ever seeing those keywords. Because the keywords meta tag was so often abused, many years ago Google began disregarding the keywords meta tag.

Google took action first. In 2009, the search engine officially announced that it does not use the meta keywords tag as a ranking factor. 

Google has just removed another feature from the Google Search Console: the content keywords report, one of the earliest features found in the Google Search Console when it was first built

So the real answer is Google and I believe Bing are now rank sites based on context rather that just a list of words and phrases.

As to using it there is no negative effect for Google. The reason most do not use it is that since Google does not, why would you put the keywords in and make it easy for your competitors to know what you are targeting?


Are you still Using Keywords?


If you really want to use the meta keywords tag, then consider only useng them sparingly. That means you need to be sure you only place the few most relevant keywords within the tag. 


So What Meta tags are there to use?


Remember there are a lot of meta tags that are still used & you must use them properly.
Here a list I can remember:



  • html tag title
  • meta name description
  • meta name robots
  • meta name revisit-after
  • meta name abstract
  • meta name author
  • meta name contact
  • meta name copyright
  • meta name distribution
  • meta name expires
  • meta name generator
  • meta name googlebot
  • meta name language
  • meta name news keywords
  • meta name no email
  • meta name rating
  • meta name reply-to
  • meta name slurp
  • meta name webauthor
     
  • meta equiv cache-control
  • meta equiv content-type
  • meta equiv cookie
  • meta equiv disposition
  • meta equiv imagetoolbar
  • meta equiv ms theme
  • meta equiv pics-label
  • meta equiv pragma
  • meta equiv refresh
  • meta equiv resource type
  • meta equiv script-type
  • meta equiv style-type
  • meta equiv window-target
  • meta data Dublin Core
  • Meta tag Rel="nofollow"
  • Meta tag Rel="canonical"
  • Miscellaneous Meta Tags
  • Miscellaneous http-equiv

How do I know what meta tags to use?

I would suggest you use a meta tag generator till you find the list of meta tags you want to use.  You should also look at a generator for your twitter card & Open Graph protocol.  If you search Google you can find a lot of  meta tag generators

There are even generators from Facebook & Twitter.

So the Answer is meta tags  are Good, but keyword meta tags are dead.