Iceland and HTTPS | BioMech XYZ

A few weeks back a comment on a programmers group on Facebook left me wondering about how many Icelandic websites really use HTTPS so I did the only logical thing, spent hours and hours finding out.

I wanted to answer 2 questions with this, which I believe I have.

1: What percentage of .is urls are using HTTPS
2: What Cert authorities are they using.

Those not interested in the method can skip “The Code” section and go straight to the graphs and data

The Code

I started by scraping some lists of .is urls and managed to get about 300 urls which I didn’t think was enough. So I ended up paying 5$ or so for a list with 17398 .is urls. Then I got to writing some software to do something with it.

Like is expected of me I tried doing it in bash and when that failed I turned to python, however I didn’t find good enough python libraries so I ended up also using a little bit of bash. I forgot to note down how long it took to run this but if I recall correctly it was around 17 hours.

I reran the code with a much larger URL set (53K) and published the raw JSON on my github

import ssl, socket
import json
import sys
import requests
import subprocess




file = "~/domaintest/list.csv"
outputfile = "~/domaintest/output.json"

f = open(file, "r")
urls = f.read().splitlines()



i = 1
tot = len(urls)


f1=open(outputfile, 'w+')


for url in urls:


        hr = ""
        sr = ""
        rd = ""
        notafter = "N/A"
        notbefore = "N/A"
        valid = "N/A"

        print str(i) + "/" + str(tot)

        jsdat = [ ]

        hostname = url

        try:
                ctx = ssl.create_default_context()
                s = ctx.wrap_socket(socket.socket(), server_hostname=hostname)
                s.settimeout(3)
                s.connect((hostname, 443))
                cert = s.getpeercert()


                subject = dict(x[0] for x in cert['subject'])
                issued_to = subject['commonName']
                issuer = dict(x[0] for x in cert['issuer'])
                issued_by = issuer['commonName']


                try:
                        process = subprocess.Popen(["~/domaintest/x.sh" , url], stdout=subprocess.PIPE)
                        output, err = process.communicate()
                        output = output.splitlines()
                        valid = output[0]
                        notbefore = output[1]
                        notafter = output[2]




                except Exception as e:
                        print(e)
                        valid = "err0r"
                        notbefore = "err0r"
                        notafter = "err0r"


        except:
                issued_by = "None"


        try:
                hr = requests.get("http://" + hostname, timeout=1)
                xrd = hr.url

                if "https" in xrd:
                        rd = True
                else:
                        rd = False


        except:
                hr = "No Response"


        try:
                sr = requests.get("https://" + hostname, timeout=1)
        except:
                sr = "No Response"


        jsdat = {


                'name'          :       url,
                'issuer'        :       issued_by,
                'http_code'     :       hr,
                'https_code'    :       sr,
                'https_redir'   :       rd,
                'notafter'      :       notafter,
                'notbefore'     :       notbefore,
                'validity'      :       valid
        }





        f1.write(str(jsdat))



        i = i + 1





f1.close()
print "all done"

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

import ssl, socket

import json

import sys

import requests

import subprocess

file = "~/domaintest/list.csv"

outputfile = "~/domaintest/output.json"

f = open(file, "r")

urls = f.read().splitlines()

i = 1

tot = len(urls)

f1=open(outputfile, 'w+')

for url in urls:

hr = ""

sr = ""

rd = ""

notafter = "N/A"

notbefore = "N/A"

valid = "N/A"

print str(i) + "/" + str(tot)

jsdat = [ ]

hostname = url

try:

ctx = ssl.create_default_context()

s = ctx.wrap_socket(socket.socket(), server_hostname=hostname)

s.settimeout(3)

s.connect((hostname, 443))

cert = s.getpeercert()

subject = dict(x[0] for x in cert['subject'])

issued_to = subject['commonName']

issuer = dict(x[0] for x in cert['issuer'])

issued_by = issuer['commonName']

try:

process = subprocess.Popen(["~/domaintest/x.sh" , url], stdout=subprocess.PIPE)

output, err = process.communicate()

output = output.splitlines()

valid = output[0]

notbefore = output[1]

notafter = output[2]

except Exception as e:

print(e)

valid = "err0r"

notbefore = "err0r"

notafter = "err0r"

except:

issued_by = "None"

try:

hr = requests.get("http://" + hostname, timeout=1)

xrd = hr.url

if "https" in xrd:

rd = True

else:

rd = False

except:

hr = "No Response"

try:

sr = requests.get("https://" + hostname, timeout=1)

except:

sr = "No Response"

jsdat = {

'name' : url,

'issuer' : issued_by,

'http_code' : hr,

'https_code' : sr,

'https_redir' : rd,

'notafter' : notafter,

'notbefore' : notbefore,

'validity' : valid

}

f1.write(str(jsdat))

i = i + 1

f1.close()

print "all done"

Those who bothered reading the code might have noticed I called a script there, x.sh

#!/bin/bash


tstamp=$(date +%s)
tmpfile="/tmp/$tstamp$$.txt"

server=$1

~/domaintest/2.sh $server &> $tmpfile


resp=$(cat $tmpfile | grep error)
respstat=$?

if [ $respstat -eq 0 ];then
        valid=$resp
else
        valid="valid"
fi

sdate=$(cat $tmpfile | grep 'Not Before')
edate=$(cat $tmpfile | grep 'Not After')

echo $valid
echo $sdate
echo $edate

rm $tmpfile

#!/bin/bash

tstamp=$(date +%s)

tmpfile="/tmp/$tstamp$$.txt"

server=$1

~/domaintest/2.sh $server &> $tmpfile

resp=$(cat $tmpfile | grep error)

respstat=$?

if [ $respstat -eq 0 ];then

valid=$resp

else

valid="valid"

sdate=$(cat $tmpfile | grep 'Not Before')

edate=$(cat $tmpfile | grep 'Not After')

echo $valid

echo $sdate

echo $edate

rm $tmpfile

and here we enter some shitty-code™ territory where this bash script calls another one, 2.sh

#!/bin/bash


server=$1




timeout 5 openssl s_client -connect $server:443 \
    -servername $server </dev/null |\
    openssl x509 -in /dev/stdin -noout -text -certopt no_header -certopt no_serial -certopt no_signame -certopt no_subject -certopt no_issuer -certopt no_pubkey -certopt no_sigdump -certopt no_aux -certopt no_extensions

#!/bin/bash

server=$1

timeout 5 openssl s_client -connect $server:443 \

-servername $server </dev/null |\

openssl x509 -in /dev/stdin -noout -text -certopt no_header -certopt no_serial -certopt no_signame -certopt no_subject -certopt no_issuer -certopt no_pubkey -certopt no_sigdump -certopt no_aux -certopt no_extensions

There is some reason for this, Python didn’t print out good enough info so I decided to use openssl, the way openssl prints out the data it was easiest to just nest it, I think. It has been about 3 weeks since I wrote the software and I of course didn’t write a single comment, maybe I was just being lazy.

The Graphs

Here we have what I thought was most interesting mixed with some contextual data.
In most cases the data-set for the graphs are pretty large and can be read by expanding the code-viewer beneath it

HTTP Response Codes

Right off the bat we see we have only 75% of 200 responses, 21% no response and a mixed bag of all sorts of responses

<Response[200]>,	13031
'NoResponse',	3657
<Response[403]>,	364
<Response[404]>,	162
<Response[503]>,	66
<Response[500]>,	48
<Response[429]>,	24
<Response[401]>,	17
<Response[502]>,	8
<Response[521]>,	7
<Response[400]>,	3
<Response[412]>,	2
<Response[424]>,	2
<Response[301]>,	1
<Response[402]>,	1
<Response[406]>,	1
<Response[409]>,	1
<Response[410]>,	1
<Response[451]>,	1
<Response[999]>,	1

<Response[200]>, 13031

'NoResponse', 3657

<Response[403]>, 364

<Response[404]>, 162

<Response[503]>, 66

<Response[500]>, 48

<Response[429]>, 24

<Response[401]>, 17

<Response[502]>, 8

<Response[521]>, 7

<Response[400]>, 3

<Response[412]>, 2

<Response[424]>, 2

<Response[301]>, 1

<Response[402]>, 1

<Response[406]>, 1

<Response[409]>, 1

<Response[410]>, 1

<Response[451]>, 1

<Response[999]>, 1

HTTPS Response Codes

'NoResponse',	10723
<Response[200]>,	6296
<Response[403]>,	179
<Response[404]>,	65
<Response[429]>,	51
<Response[503]>,	34
<Response[500]>,	17
<Response[521]>,	10
<Response[401]>,	6
<Response[525]>,	5
<Response[424]>,	2
<Response[502]>,	2
<Response[301]>,	1
<Response[400]>,	1
<Response[402]>,	1
<Response[406]>,	1
<Response[451]>,	1
<Response[523]>,	1
<Response[526]>,	1
<Response[999]>,	1

'NoResponse', 10723

<Response[200]>, 6296

<Response[403]>, 179

<Response[404]>, 65

<Response[429]>, 51

<Response[503]>, 34

<Response[500]>, 17

<Response[521]>, 10

<Response[401]>, 6

<Response[525]>, 5

<Response[424]>, 2

<Response[502]>, 2

<Response[301]>, 1

<Response[400]>, 1

<Response[402]>, 1

<Response[406]>, 1

<Response[451]>, 1

<Response[523]>, 1

<Response[526]>, 1

<Response[999]>, 1

HTTPS usage on .is urls

No SSL	9764
SSL	7634

.is HTTPS providers

Data in graph:

Lets Encrypt Authority X3	4669
cPanel	1600
COMODOECCDomainValidationSecureServerCA2	550
COMODORSADomainValidationSecureServerCA	170
Other	125
RapidSSLSHA256CA	71
GoDaddySecureCertificateAuthority-G2	69
DigiCertSHA2SecureServerCA	63
DigiCertSHA2HighAssuranceServerCA	59
RapidSSLRSACA2018	57
Amazon	45
GlobalSignOrganizationValidationCA-SHA256-G2	36
DigiCertSHA2ExtendedValidationServerCA	21
AlphaSSLCA-SHA256-G2	19
GlobalSignDomainValidationCA-SHA256-G2	19
UbiquiTLS\u2122DVRSAServerCA	15
GeoTrustDVSSLCA-G3	14
RapidSSLTLSRSACAG1	11
StarfieldSecureCertificateAuthority-G2	11
CloudFlareIncECCCA-2	10

Lets Encrypt Authority X3 4669

cPanel 1600

COMODOECCDomainValidationSecureServerCA2 550

COMODORSADomainValidationSecureServerCA 170

Other 125

RapidSSLSHA256CA 71

GoDaddySecureCertificateAuthority-G2 69

DigiCertSHA2SecureServerCA 63

DigiCertSHA2HighAssuranceServerCA 59

RapidSSLRSACA2018 57

Amazon 45

GlobalSignOrganizationValidationCA-SHA256-G2 36

DigiCertSHA2ExtendedValidationServerCA 21

AlphaSSLCA-SHA256-G2 19

GlobalSignDomainValidationCA-SHA256-G2 19

UbiquiTLS\u2122DVRSAServerCA 15

GeoTrustDVSSLCA-G3 14

RapidSSLTLSRSACAG1 11

StarfieldSecureCertificateAuthority-G2 11

CloudFlareIncECCCA-2 10

Data with “Other” expanded

u"Let's Encrypt Authority X3"	4669
u'cPanel,	1600
u'COMODOECCDomainValidationSecureServerCA2'	550
u'COMODORSADomainValidationSecureServerCA'	170
u'RapidSSLSHA256CA'	71
u'GoDaddySecureCertificateAuthority-G2'	69
u'DigiCertSHA2SecureServerCA'	63
u'DigiCertSHA2HighAssuranceServerCA'	59
u'RapidSSLRSACA2018'	57
u'Amazon'	45
u'GlobalSignOrganizationValidationCA-SHA256-G2'	36
u'DigiCertSHA2ExtendedValidationServerCA'	21
u'AlphaSSLCA-SHA256-G2'	19
u'GlobalSignDomainValidationCA-SHA256-G2'	19
u'UbiquiTLS\u2122DVRSAServerCA'	15
u'GeoTrustDVSSLCA-G3'	14
u'RapidSSLTLSRSACAG1'	11
u'StarfieldSecureCertificateAuthority-G2'	11
u'CloudFlareIncECCCA-2'	10
u'RapidSSLSHA256CA-G3'	9
u'GeoTrustTLSRSACAG1'	8
u'GoogleInternetAuthorityG3'	8
u'thawteDVSSLSHA256CA'	8
u'COMODORSAOrganizationValidationSecureServerCA'	7
u'GlobalSignExtendedValidationCA-SHA256-G3'	7
u'RapidSSLSHA256CA-G2'	7
u'GeoTrustSSLCA-G3'	6
u'SSL.comDVCA'	6
u'thawteSSLCA-G2'	5
u'GeoTrustRSACA2018'	4
u'GlobalSignCloudSSLCA-SHA256-G3'	4
u'GlobalSignExtendedValidationCA-SHA256-G2'	4
u'ThawteTLSRSACAG1'	4
u'COMODORSAExtendedValidationSecureServerCA'	3
u'GeoTrustSHA256SSLCA'	3
u'EncryptionEverywhereDVTLSCA-G2'	2
u'EntrustCertificationAuthority-L1K'	2
u'GeoTrustDVSSLSHA256CA'	2
u'GeoTrustEVSSLCA-G4'	2
u'GlobalSignDomainValidationCA-SHA256-G3'	2
u'RapidSSLSHA256CA-G4'	2
u'SymantecBasicDVSSLCA-G2'	2
u'SymantecClass3SecureServerCA-G4'	2
u'TERENASSLHighAssuranceCA3'	2
u'thawteEVSSLCA-G3'	2
u'ThawteRSACA2018'	2
u'TrustedSecureCertificateAuthority5'	2
u'GandiStandardSSLCA2'	1
u'GeoTrustEVRSACA2018'	1
u'MicrosoftITSSLSHA2'	1
u'SecureCoreRSADVCA'	1
u'thawteExtendedValidationSHA256SSLCA'	1
u'TrustedSecureCertificateAuthorityDV'	1
u'TrusticoRSADVCA'	1
u'TrustProviderB.V.DVSSLCA-G2'	1

u"Let's Encrypt Authority X3" 4669

u'cPanel, 1600

u'COMODOECCDomainValidationSecureServerCA2' 550

u'COMODORSADomainValidationSecureServerCA' 170

u'RapidSSLSHA256CA' 71

u'GoDaddySecureCertificateAuthority-G2' 69

u'DigiCertSHA2SecureServerCA' 63

u'DigiCertSHA2HighAssuranceServerCA' 59

u'RapidSSLRSACA2018' 57

u'Amazon' 45

u'GlobalSignOrganizationValidationCA-SHA256-G2' 36

u'DigiCertSHA2ExtendedValidationServerCA' 21

u'AlphaSSLCA-SHA256-G2' 19

u'GlobalSignDomainValidationCA-SHA256-G2' 19

u'UbiquiTLS\u2122DVRSAServerCA' 15

u'GeoTrustDVSSLCA-G3' 14

u'RapidSSLTLSRSACAG1' 11

u'StarfieldSecureCertificateAuthority-G2' 11

u'CloudFlareIncECCCA-2' 10

u'RapidSSLSHA256CA-G3' 9

u'GeoTrustTLSRSACAG1' 8

u'GoogleInternetAuthorityG3' 8

u'thawteDVSSLSHA256CA' 8

u'COMODORSAOrganizationValidationSecureServerCA' 7

u'GlobalSignExtendedValidationCA-SHA256-G3' 7

u'RapidSSLSHA256CA-G2' 7

u'GeoTrustSSLCA-G3' 6

u'SSL.comDVCA' 6

u'thawteSSLCA-G2' 5

u'GeoTrustRSACA2018' 4

u'GlobalSignCloudSSLCA-SHA256-G3' 4

u'GlobalSignExtendedValidationCA-SHA256-G2' 4

u'ThawteTLSRSACAG1' 4

u'COMODORSAExtendedValidationSecureServerCA' 3

u'GeoTrustSHA256SSLCA' 3

u'EncryptionEverywhereDVTLSCA-G2' 2

u'EntrustCertificationAuthority-L1K' 2

u'GeoTrustDVSSLSHA256CA' 2

u'GeoTrustEVSSLCA-G4' 2

u'GlobalSignDomainValidationCA-SHA256-G3' 2

u'RapidSSLSHA256CA-G4' 2

u'SymantecBasicDVSSLCA-G2' 2

u'SymantecClass3SecureServerCA-G4' 2

u'TERENASSLHighAssuranceCA3' 2

u'thawteEVSSLCA-G3' 2

u'ThawteRSACA2018' 2

u'TrustedSecureCertificateAuthority5' 2

u'GandiStandardSSLCA2' 1

u'GeoTrustEVRSACA2018' 1

u'MicrosoftITSSLSHA2' 1

u'SecureCoreRSADVCA' 1

u'thawteExtendedValidationSHA256SSLCA' 1

u'TrustedSecureCertificateAuthorityDV' 1

u'TrusticoRSADVCA' 1

u'TrustProviderB.V.DVSSLCA-G2' 1

Summary:

Not sure what to summarize, the data speaks for itself. It seems that Iceland runs on Let’s Encrypt, just like this blog

BioMech XYZ

The Code

The Graphs

HTTP Response Codes

HTTPS Response Codes

Summary:

Leave a Reply Cancel reply