Playing around from where I left off last time, the Canadian Securities Administrators (“CSA”) – Disciplined Persons site has loads of information available about disciplined registrants (those registered with securities commissions in one of the 10 provinces and 3 territories within Canada and licensed to sell to and advise Canadians in regards to securities products).
The data goes back a bunch of years, but short of ‘hunting-and-pecking’, name-by-name, the site does not give any overviews on what is going on with ‘Disciplined Persons’. Aggregating the data using a Ruby crawler and “R” to graph the result offers a better view.
Here’s a Treemap showing disciplinary orders issued by various provincial commissions\court:
The next Treemap shows the aggregate data for the penalties meted out by Canadian securities regulators and courts in connection with orders relating securities regulations infractions :
Take special note of 41 disciplined-persons penalized with imprisonment!
Here’s the code to crawl the data:
require 'nokogiri' require 'rubygems' require 'watir-webdriver'
browser = Watir::Browser.new :chrome browser.goto 'https://www.securities-administrators.ca/disciplinedpersons.aspx?id=74'
seen = Array.new
('A'..'Z').each do |letter|
browser.link(:text => letter).click browser.table(:id => 'ctl00_bodyContent_gv_list').wait_until_present
loop do table_Rows = browser.table(:id, 'ctl00_bodyContent_gv_list').rows.length
for i in 1..table_Rows.to_i-1 name = browser.table(:id, 'ctl00_bodyContent_gv_list')[i][0].text date_of_order = browser.table(:id, 'ctl00_bodyContent_gv_list')[i][1].text browser.link(:text, name).click browser.span(:id => 'ctl00_bodyContent_lbl_persondetail').wait_until_present
discipline_table_Rows = browser.table(:id, 'ctl00_bodyContent_DataList1').rows.length
for t in 0..discipline_table_Rows.to_i - 1
doc = Nokogiri::HTML.parse(browser.html)
t_child = t.to_i + 1
if t < 1 && discipline_table_Rows < 2 ### Only one finding ###
date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr > td > table > tbody > tr:nth-child(1) > td').text
$regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl00_lbl_juridiction').div.text
payment = doc.css('#ctl00_bodyContent_DataList1_ctl00_tr_SanctionList+ tr td').text
sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl00_lbl_sanction div').text
violations = doc.css('#ctl00_bodyContent_DataList1_ctl00_lbl_violation div').text
elsif t < 2 ### More than one finding ###
date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr:nth-child('+t_child.to_s+') > td > table > tbody > tr:nth-child(1) > td').text
$regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl00_lbl_juridiction').div(:class, 'sectiontitle').text
payment = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_tr_SanctionList+ tr td').text
sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_sanction div').text
violations = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_violation div').text
else
date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr:nth-child('+t_child.to_s+') > td > table > tbody > tr:nth-child(1) > td').text
if browser.table(:id , 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_juridiction').div(:class, 'sectiontitle').exists?
$regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id , 'ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_juridiction').div(:class, 'sectiontitle').text
else
end
payment = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_tr_SanctionList+ tr td').text
sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_sanction div').text
violations = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_violation div').text
end
if (seen).include?("#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}")
else
puts "#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}"
end
seen << "#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}"
end
browser.back browser.span(:id => 'ctl00_bodyContent_lbl_browse').wait_until_present end
break if browser.div(:id, 'ctl00_bodyContent_pnl_pager_top').text !~ /\d+\.\.\./ browser.link(:id, 'ctl00_bodyContent_lbtnNext').click sleep (2) browser.table(:id => 'ctl00_bodyContent_gv_list').wait_until_present
end
end