Iceland and HTTPS

A few weeks back a comment on a programmers group on Facebook left me wondering about how many Icelandic websites really use HTTPS so I did the only logical thing, spent hours and hours finding out.
I wanted to answer 2 questions with this, which I believe I have.
1: What percentage of .is urls are using HTTPS
2: What Cert authorities are they using.
Those not interested in the method can skip “The Code” section and go straight to the graphs and data
The Code
I started by scraping some lists of .is urls and managed to get about 300 urls which I didn’t think was enough. So I ended up paying 5$ or so for a list with 17398 .is urls. Then I got to writing some software to do something with it.
Like is expected of me I tried doing it in bash and when that failed I turned to python, however I didn’t find good enough python libraries so I ended up also using a little bit of bash. I forgot to note down how long it took to run this but if I recall correctly it was around 17 hours.
I reran the code with a much larger URL set (53K) and published the raw JSON on my github
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
import ssl, socket import json import sys import requests import subprocess file = "~/domaintest/list.csv" outputfile = "~/domaintest/output.json" f = open(file, "r") urls = f.read().splitlines() i = 1 tot = len(urls) f1=open(outputfile, 'w+') for url in urls: hr = "" sr = "" rd = "" notafter = "N/A" notbefore = "N/A" valid = "N/A" print str(i) + "/" + str(tot) jsdat = [ ] hostname = url try: ctx = ssl.create_default_context() s = ctx.wrap_socket(socket.socket(), server_hostname=hostname) s.settimeout(3) s.connect((hostname, 443)) cert = s.getpeercert() subject = dict(x[0] for x in cert['subject']) issued_to = subject['commonName'] issuer = dict(x[0] for x in cert['issuer']) issued_by = issuer['commonName'] try: process = subprocess.Popen(["~/domaintest/x.sh" , url], stdout=subprocess.PIPE) output, err = process.communicate() output = output.splitlines() valid = output[0] notbefore = output[1] notafter = output[2] except Exception as e: print(e) valid = "err0r" notbefore = "err0r" notafter = "err0r" except: issued_by = "None" try: hr = requests.get("http://" + hostname, timeout=1) xrd = hr.url if "https" in xrd: rd = True else: rd = False except: hr = "No Response" try: sr = requests.get("https://" + hostname, timeout=1) except: sr = "No Response" jsdat = { 'name' : url, 'issuer' : issued_by, 'http_code' : hr, 'https_code' : sr, 'https_redir' : rd, 'notafter' : notafter, 'notbefore' : notbefore, 'validity' : valid } f1.write(str(jsdat)) i = i + 1 f1.close() print "all done" |
Those who bothered reading the code might have noticed I called a script there, x.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#!/bin/bash tstamp=$(date +%s) tmpfile="/tmp/$tstamp$$.txt" server=$1 ~/domaintest/2.sh $server &> $tmpfile resp=$(cat $tmpfile | grep error) respstat=$? if [ $respstat -eq 0 ];then valid=$resp else valid="valid" fi sdate=$(cat $tmpfile | grep 'Not Before') edate=$(cat $tmpfile | grep 'Not After') echo $valid echo $sdate echo $edate rm $tmpfile |
and here we enter some shitty-code™ territory where this bash script calls another one, 2.sh
1 2 3 4 5 6 7 8 9 10 11 |
#!/bin/bash server=$1 timeout 5 openssl s_client -connect $server:443 \ -servername $server </dev/null |\ openssl x509 -in /dev/stdin -noout -text -certopt no_header -certopt no_serial -certopt no_signame -certopt no_subject -certopt no_issuer -certopt no_pubkey -certopt no_sigdump -certopt no_aux -certopt no_extensions |
There is some reason for this, Python didn’t print out good enough info so I decided to use openssl, the way openssl prints out the data it was easiest to just nest it, I think. It has been about 3 weeks since I wrote the software and I of course didn’t write a single comment, maybe I was just being lazy.
The Graphs
Here we have what I thought was most interesting mixed with some contextual data.
In most cases the data-set for the graphs are pretty large and can be read by expanding the code-viewer beneath it
HTTP Response Codes
Right off the bat we see we have only 75% of 200 responses, 21% no response and a mixed bag of all sorts of responses
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
<Response[200]>, 13031 'NoResponse', 3657 <Response[403]>, 364 <Response[404]>, 162 <Response[503]>, 66 <Response[500]>, 48 <Response[429]>, 24 <Response[401]>, 17 <Response[502]>, 8 <Response[521]>, 7 <Response[400]>, 3 <Response[412]>, 2 <Response[424]>, 2 <Response[301]>, 1 <Response[402]>, 1 <Response[406]>, 1 <Response[409]>, 1 <Response[410]>, 1 <Response[451]>, 1 <Response[999]>, 1 |
HTTPS Response Codes

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
'NoResponse', 10723 <Response[200]>, 6296 <Response[403]>, 179 <Response[404]>, 65 <Response[429]>, 51 <Response[503]>, 34 <Response[500]>, 17 <Response[521]>, 10 <Response[401]>, 6 <Response[525]>, 5 <Response[424]>, 2 <Response[502]>, 2 <Response[301]>, 1 <Response[400]>, 1 <Response[402]>, 1 <Response[406]>, 1 <Response[451]>, 1 <Response[523]>, 1 <Response[526]>, 1 <Response[999]>, 1 |
HTTPS usage on .is urls
No SSL | 9764 |
SSL | 7634 |
.is HTTPS providers
Data in graph:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
Lets Encrypt Authority X3 4669 cPanel 1600 COMODOECCDomainValidationSecureServerCA2 550 COMODORSADomainValidationSecureServerCA 170 Other 125 RapidSSLSHA256CA 71 GoDaddySecureCertificateAuthority-G2 69 DigiCertSHA2SecureServerCA 63 DigiCertSHA2HighAssuranceServerCA 59 RapidSSLRSACA2018 57 Amazon 45 GlobalSignOrganizationValidationCA-SHA256-G2 36 DigiCertSHA2ExtendedValidationServerCA 21 AlphaSSLCA-SHA256-G2 19 GlobalSignDomainValidationCA-SHA256-G2 19 UbiquiTLS\u2122DVRSAServerCA 15 GeoTrustDVSSLCA-G3 14 RapidSSLTLSRSACAG1 11 StarfieldSecureCertificateAuthority-G2 11 CloudFlareIncECCCA-2 10 |
Data with “Other” expanded
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
u"Let's Encrypt Authority X3" 4669 u'cPanel, 1600 u'COMODOECCDomainValidationSecureServerCA2' 550 u'COMODORSADomainValidationSecureServerCA' 170 u'RapidSSLSHA256CA' 71 u'GoDaddySecureCertificateAuthority-G2' 69 u'DigiCertSHA2SecureServerCA' 63 u'DigiCertSHA2HighAssuranceServerCA' 59 u'RapidSSLRSACA2018' 57 u'Amazon' 45 u'GlobalSignOrganizationValidationCA-SHA256-G2' 36 u'DigiCertSHA2ExtendedValidationServerCA' 21 u'AlphaSSLCA-SHA256-G2' 19 u'GlobalSignDomainValidationCA-SHA256-G2' 19 u'UbiquiTLS\u2122DVRSAServerCA' 15 u'GeoTrustDVSSLCA-G3' 14 u'RapidSSLTLSRSACAG1' 11 u'StarfieldSecureCertificateAuthority-G2' 11 u'CloudFlareIncECCCA-2' 10 u'RapidSSLSHA256CA-G3' 9 u'GeoTrustTLSRSACAG1' 8 u'GoogleInternetAuthorityG3' 8 u'thawteDVSSLSHA256CA' 8 u'COMODORSAOrganizationValidationSecureServerCA' 7 u'GlobalSignExtendedValidationCA-SHA256-G3' 7 u'RapidSSLSHA256CA-G2' 7 u'GeoTrustSSLCA-G3' 6 u'SSL.comDVCA' 6 u'thawteSSLCA-G2' 5 u'GeoTrustRSACA2018' 4 u'GlobalSignCloudSSLCA-SHA256-G3' 4 u'GlobalSignExtendedValidationCA-SHA256-G2' 4 u'ThawteTLSRSACAG1' 4 u'COMODORSAExtendedValidationSecureServerCA' 3 u'GeoTrustSHA256SSLCA' 3 u'EncryptionEverywhereDVTLSCA-G2' 2 u'EntrustCertificationAuthority-L1K' 2 u'GeoTrustDVSSLSHA256CA' 2 u'GeoTrustEVSSLCA-G4' 2 u'GlobalSignDomainValidationCA-SHA256-G3' 2 u'RapidSSLSHA256CA-G4' 2 u'SymantecBasicDVSSLCA-G2' 2 u'SymantecClass3SecureServerCA-G4' 2 u'TERENASSLHighAssuranceCA3' 2 u'thawteEVSSLCA-G3' 2 u'ThawteRSACA2018' 2 u'TrustedSecureCertificateAuthority5' 2 u'GandiStandardSSLCA2' 1 u'GeoTrustEVRSACA2018' 1 u'MicrosoftITSSLSHA2' 1 u'SecureCoreRSADVCA' 1 u'thawteExtendedValidationSHA256SSLCA' 1 u'TrustedSecureCertificateAuthorityDV' 1 u'TrusticoRSADVCA' 1 u'TrustProviderB.V.DVSSLCA-G2' 1 |
Summary:
Not sure what to summarize, the data speaks for itself. It seems that Iceland runs on Let’s Encrypt, just like this blog