{"id":275,"date":"2024-06-28T10:45:02","date_gmt":"2024-06-28T09:45:02","guid":{"rendered":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/?p=275"},"modified":"2024-07-01T11:44:34","modified_gmt":"2024-07-01T10:44:34","slug":"high-speed-rail-data-querying-train-prices-with-c","status":"publish","type":"post","link":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/2024\/06\/28\/high-speed-rail-data-querying-train-prices-with-c\/","title":{"rendered":"High Speed Rail (Data): Querying Train Prices with C++"},"content":{"rendered":"<p>Hi, I\u2019m Hamish; I\u2019m from Watford, UK, and I\u2019m in the second year of Computing. In my free time I enjoy studying foreign languages, playing water polo and doing Competitive Programming. This year I\u2019m behind Imperial CyberSoc and CTF Team. Let\u2019s connect on\u00a0<a href=\"https:\/\/www.linkedin.com\/in\/hamish-starling-147859235\/\">LinkedIn<\/a>!<img loading=\"lazy\" decoding=\"async\" width=\"167\" height=\"205\" class=\"alignright wp-image-256 size-full\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2023\/11\/Hamish-photo.png\" alt=\"Hamish photo\" \/><\/p>\n<hr \/>\n<p><span data-contrast=\"none\"><span class=\"TextRun SCXW37049089 BCX0\" lang=\"EN-GB\" xml:lang=\"EN-GB\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW37049089 BCX0\">A<\/span> <span class=\"NormalTextRun SCXW37049089 BCX0\">few months <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">ago,<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\"> I took <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">the <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">train to Oxford with a <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">school-<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">friend. <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">Rolling through the <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">picturesque <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">Chilterns<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">, <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">the sun <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">streaming through the <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">window<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">,<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\"> and as the fans of <\/span><\/span><a class=\"Hyperlink SCXW37049089 BCX0\" href=\"https:\/\/en.wikipedia.org\/wiki\/Great_British_Railway_Journeys\" target=\"_blank\" rel=\"noreferrer noopener\"><span class=\"TextRun Underlined SCXW37049089 BCX0\" lang=\"EN-GB\" xml:lang=\"EN-GB\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW37049089 BCX0\" data-ccp-charstyle=\"Hyperlink\">GBRJ<\/span><\/span><\/a><span class=\"TextRun SCXW37049089 BCX0\" lang=\"EN-GB\" xml:lang=\"EN-GB\" data-contrast=\"auto\"> <span class=\"NormalTextRun SCXW37049089 BCX0\">that we are, we proposed<\/span> <span class=\"NormalTextRun SCXW37049089 BCX0\">visit<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">ing more <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">UK cities by train<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">. Yet, existing booking sites <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">wouldn\u2019t<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\"> let us sort destinations by price.<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\"> Fin<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">e<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">, we thought: a bit of programming <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">will <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">solve this<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">. <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">It also <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">happened to <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">explain a<\/span> <span class=\"NormalTextRun SCXW37049089 BCX0\">long-running enigma <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">&#8211; <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">why <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">UK <\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">train prices seem so random<\/span><span class=\"NormalTextRun SCXW37049089 BCX0\">:<\/span><\/span><span class=\"EOP SCXW37049089 BCX0\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/span><\/p>\n<p><!--more--><\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"768\" class=\"wp-image-276 size-large aligncenter\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2024\/06\/Chiltern1-1024x768.jpeg\" alt=\"Hamish smiling infront of a parked train with both thumbs up\" \/> <img loading=\"lazy\" decoding=\"async\" width=\"933\" height=\"703\" class=\"wp-image-277 aligncenter\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2024\/06\/Chiltern-2.png\" alt=\"\" \/><\/p>\n<p><i><span data-contrast=\"auto\">Thanks to Chiltern Railways for (not!) sponsoring this video\u2026<\/span><\/i><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/opendata.nationalrail.co.uk\/\"><span data-contrast=\"none\">National Rail<\/span><\/a><span data-contrast=\"auto\"> publishes open fare data, and various programs have been distributed to process it, from <\/span><a href=\"https:\/\/librailfare.sourceforge.net\/\"><span data-contrast=\"none\">librailfare<\/span><\/a><span data-contrast=\"auto\"> and Avantix Travelller (old data format) to BR Fares (web API unsuitable for our query volume).\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">The most promising implementation I found was @FedericoBotta\u2019s <\/span><a href=\"https:\/\/github.com\/FedericoBotta\/railfares\"><span data-contrast=\"none\">railfares<\/span><\/a><span data-contrast=\"auto\"> which, although using the new data, computes \u2018isocosts\u2019 rather than journey costs. It therefore didn\u2019t seem to consider \u2018split-ticketing\u2019, changing train, or non-reversible fares. I also struggled to make it run quickly on my computer: problematic when examining fares from all tube-accessible stations. Time, then, to write some C++.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Things started off swimmingly with some rudimentary code to import the data, and I ran Dijkstra\u2019s algorithm to compute the cost to every station with unlimited changes. This gave interesting results. I could reach most UK stations for less than \u00a320, even Glasgow for only \u00a314 one-way:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"384\" height=\"140\" class=\"size-full wp-image-278 aligncenter\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2024\/06\/Route.png\" alt=\"\" \/><\/p>\n<p><span data-contrast=\"auto\">But closer review revealed a snag: my access to virtually all Northern stations hinged on this mysterious \u2018NXP\u2019 ticket to Retford for only \u00a38. The wonderful <\/span><a href=\"https:\/\/www.brfares.com\/\"><span data-contrast=\"none\">BRFares<\/span><\/a><span data-contrast=\"auto\"> identified it as LNER\u2019s \u201cflat fare\u201d advance, but alas on their website it was nowhere to be booked. A <\/span><a href=\"https:\/\/x.com\/LNER\/status\/1295784297627037697\"><span data-contrast=\"none\">twitter response<\/span><\/a><span data-contrast=\"auto\"> from customer support confirmed that these are special tickets in limited numbers (often 0!).<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">This proved a recurring theme as I tried the tickets my program proposed. Almost all cheap rail tickets are quota-controlled \u201cAdvance\u201d fares. While commonly available, they typically have &gt;20 different price points and without the quota (RARS) data, choosing the correct one is impossible.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">After much frustration, I resigned to only Anytime and Off-Peak tickets. These are so expensive they are useless, but at least I could check the results:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"797\" height=\"698\" class=\"wp-image-279 aligncenter\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2024\/06\/map.png\" alt=\"\" \/><\/p>\n<p><i><span data-contrast=\"auto\">Furthest stations from London for \u00a3&lt;20, only \u201cwalk up\u201d fares, unlimited changes.<\/span><\/i><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Another issue with my first attempt was the runtime (&gt;25s). Even if my program didn\u2019t compute quite what I wanted, I could still optimise it to learn from it.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">In the end I reduced it to &lt;1s by:\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"-\" data-font=\"Aptos\" data-listid=\"3\" data-list-defn-props=\"{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Aptos&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"0\" data-aria-level=\"1\"><span data-contrast=\"auto\">Pruning NLCs<span style=\"font-size: 50%;vertical-align: super\">1<\/span> <\/span><\/li>\n<\/ul>\n<ul>\n<li data-leveltext=\"-\" data-font=\"Aptos\" data-listid=\"3\" data-list-defn-props=\"{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Aptos&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Turning on -Ofast (<\/span><a href=\"https:\/\/nor-blog.codeberg.page\/posts\/2021-10-26-gcc-optimization-pragmas\/\"><span data-contrast=\"none\">controversial<\/span><\/a><span data-contrast=\"auto\">\u2026 but pretty safe) <\/span><b><i><span data-contrast=\"auto\">(10s)<\/span><\/i><\/b><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li data-leveltext=\"-\" data-font=\"Aptos\" data-listid=\"3\" data-list-defn-props=\"{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Aptos&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><span data-contrast=\"auto\">Manual (not mktime) date parsing (surprising) <\/span><b><span data-contrast=\"auto\">(<\/span><\/b><b><i><span data-contrast=\"auto\">2s<\/span><\/i><\/b><b><span data-contrast=\"auto\">)<\/span><\/b><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li data-leveltext=\"-\" data-font=\"Aptos\" data-listid=\"3\" data-list-defn-props=\"{&quot;335551671&quot;:0,&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Aptos&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;-&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"3\" data-aria-level=\"1\"><span data-contrast=\"auto\">Storing ticket codes as char[][][] instead of vector&lt;vector&lt;string&gt;&gt; (also surprising: usually vector and string are fast, though 10M iterations, -0.1 \u03bcs \/ iteration saves 1s) <\/span><b><i><span data-contrast=\"auto\">(&lt; 1s)<\/span><\/i><\/b><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">I also tried memory mapping (slower) and profiling (didn\u2019t help: most code was I\/O or DLLs).<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"66\" class=\"wp-image-280 aligncenter\" src=\"https:\/\/blogs-staging.imperial.ac.uk\/computing\/files\/2024\/06\/benchmark-1024x66.png\" alt=\"\" \/><\/p>\n<p><span data-contrast=\"auto\">In conclusion, we failed \u2013 without accurate Advance quotas we still don\u2019t know where to take our next rail journey! But this was still worthwhile because it <\/span><i><span data-contrast=\"auto\">has<\/span><\/i><span data-contrast=\"auto\"> inspired me to take Performance Engineering next year.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">I also hope it makes the case for Open Data to give consumers the flexibility of their own use cases. It\u2019s been 19 years since <\/span><a href=\"https:\/\/publications.parliament.uk\/pa\/cm200506\/cmselect\/cmtran\/700\/70008.htm\"><span data-contrast=\"none\">this report<\/span><\/a><span data-contrast=\"auto\"> suggested a more transparent quota system, so I\u2019m doubtful it will happen, but hopefully projects like this show its potential.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">This was my last post for AY 23\/24 \u2013 I really hope you enjoyed them and have a great summer!\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><i><span data-contrast=\"auto\">Full code: <\/span><\/i><a href=\"https:\/\/github.com\/starswap\/trainpricing\/\"><i><span data-contrast=\"none\">https:\/\/github.com\/starswap\/trainpricing\/<\/span><\/i><\/a><i><span data-contrast=\"auto\">.<\/span><\/i><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:279}\">\u00a0<\/span><span data-contrast=\"auto\"><span style=\"font-size: 50%;vertical-align: super\">1<\/span> <\/span>National Location Codes identify stations and other features on the rail network. I removed all irrelevant non-station ones.<span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi, I\u2019m Hamish; I\u2019m from Watford, UK, and I\u2019m in the second year of Computing. In my free time I enjoy studying foreign languages, playing water polo and doing Competitive Programming. This year I\u2019m behind Imperial CyberSoc and CTF Team. Let\u2019s connect on\u00a0LinkedIn! A few months ago, I took the train to Oxford with a [&hellip;]<\/p>\n","protected":false},"author":1399,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48173],"tags":[],"class_list":["post-275","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/posts\/275","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/users\/1399"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/comments?post=275"}],"version-history":[{"count":8,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/posts\/275\/revisions"}],"predecessor-version":[{"id":290,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/posts\/275\/revisions\/290"}],"wp:attachment":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/media?parent=275"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/categories?post=275"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/computing\/wp-json\/wp\/v2\/tags?post=275"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}