B)<\/strong> You can collect the data for yourself. (E.g. via a javascript code snippet that sends data to a .csv plain text file on your server.) It’s a bit more difficult to implement since it requires some coding skills. But in the long term this solution will serve you much better (and it will be more profitable, too) than version A. Either way you choose, it’s worth understanding how raw data collection works in general — and how you can collect data from your website visitors’ behaviour.<\/strong><\/p>\n\n\n\n Do it for yourself or using a 3rd party tool\u2026 very similar things are happening under the hood!<\/p>\n\n\n\n Let’s go with the simplest example!<\/p>\n\n\n\n You have a website and you would like to collect every visitor mouse click for an upcoming data science<\/a> project.<\/p>\n\n\n\n How do you do that?<\/p>\n\n\n\n First, implement an invisible tracking script (aka “data collection script”) on every clickable element of your site! From that point on, when a website visitor clicks on a specific element (e.g. a link or a button), the click causes two things to happen:<\/p>\n\n\n\n As simple as that.<\/p>\n\n\n\n You could track every user interaction (let’s call them “events<\/em>“) on your website (or in your mobile app): page views, feature usage, clicks, taps — even mouse movements, if you need to.<\/p>\n\n\n\n A more general illustration to help you imagine what’s happening here:<\/p>\n\n\n\n When the collected raw data hits your data warehouse, it can be stored in different formats.<\/p>\n\n\n\n For startups the best format is the <\/strong>plain text<\/strong><\/em> format as it is very flexible.<\/strong> You can imagine this as a simple .txt, .csv or .tsv<\/em> file with text in it. Many companies follow this model.<\/p>\n\n\n\n But it’s also worth mentioning that many other companies (e.g. almost all multinational companies) like to collect their data directly into SQL databases (or to other similar structured formats). <\/p>\n\n\n\n And there are several other ways to store your data. (Graph databases, noSQL databases, etc.)<\/p>\n\n\n\n In this example I’ll keep it simple and will go with the most common solution: plain text format.<\/strong><\/p>\n\n\n\n Remember that each event<\/em> from a website visitor (e.g. a click on your website) creates one line<\/em> of data using your previously implemented data collection scripts. This line<\/em> goes into a file on your data server. We call this file with a bunch of events<\/em> in it a log<\/em>. You can have more than one log, but almost all of them will have the same format. Something like this:<\/p>\n\n\n\n Look messy?<\/p>\n\n\n\n Maybe at first, but go through that column by column! (This is a .csv file which means that the field-separator is a semicolon.)<\/p>\n\n\n\n These are the most basic data points that every event log should contain.<\/p>\n\n\n\n But you can add even more dimensions. Just a few examples:<\/p>\n\n\n\n If you run an online business, you can collect and store a virtually infinite amount of data. Infinite vertically (the number of different events you can log) as well as horizontally (the number of dimensions you can collect about one event in one line).<\/p>\n\n\n\n This raises the obvious question: what you should collect and what you shouldn’t.<\/strong><\/p>\n\n\n\n The principle here is very simple:<\/strong> collect everything you can.<\/strong> Every click, every pageview, every feature usage, everything.<\/p>\n\n\n\n It’s interesting to note that (according to market benchmarks) most startups who follow this collect-everything-principle actually end up using less than 10% of their data. 90% is not even touched by their data scientists!<\/p>\n\n\n\n So why do they collect everything? The answer is: because you can never know what data you will need in the future for your data projects.<\/strong><\/p>\n\n\n\n Let’s say you want to change a 3-year-old key feature of your online product. You don’t want to mess anything up, so before the change, you will spend some time to understand the exact role of that 3-year-old key feature. For that you will need to analyze your data retrospectively. Whoops, you realize that you didn’t collect any data about it. Game over, you’ve just lost 3-years worth of information\u2026 Get it?<\/p>\n\n\n\n If you start thinking about collecting a specific data point when it’s actually needed for a data science project, you are already too late.<\/strong><\/p>\n\n\n\n And that’s the reason behind the principle “collect everything you can”.<\/p>\n\n\n\n There are some obvious limitations, of course.<\/p>\n\n\n\n But the price of storing data is not one of those.<\/strong> Storing data (in the cloud at least) is very cheap today.<\/p>\n\n\n\n The real limitations are:<\/p>\n\n\n\n This is how raw data collection works at a high level. Google Analytics, Mixpanel, Crazyegg or your own data warehouses — all are based on these principles. Of course there are small differences, but now you understand what happens in the background and you can be more confident when talking about raw data collection with your co-workers!<\/p>\n\n\n\n Cheers,<\/em> Cheers,<\/em> The first step for every data science project is data collection, that is, getting the actual raw data. There are two ways to do this: A) You can pick one or more “smart tools” to use. These services will collect the data for you automatically. You only need to copy-paste a snippet of code into […]<\/p>\n","protected":false},"author":1,"featured_media":1421,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[141],"tags":[14,53,13,73,28,59,41,62,16,30],"class_list":["post-374","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-strategy-business-data-science-analytics","tag-big-data","tag-collect","tag-data","tag-data-collection","tag-google-analytics","tag-hotjar","tag-metrics","tag-mixpanel","tag-startup","tag-tomi-mester"],"yoast_head":"\n
Why? For several reasons that I’ve already written about in this article<\/a>. But here’s a quick summary:<\/p>\n\n\n\nHow does data collection work?<\/strong><\/h2>\n\n\n\n
![\"data](\"https:\/\/data36.com\/wp-content\/uploads\/2016\/11\/data-collection-1-1024x430.png\")
![\"data](\"https:\/\/data36.com\/wp-content\/uploads\/2019\/04\/data-collection-scripts-send-data-from-the-front-end-to-production-and-data-servers-1024x767.png\")
How to store the collected raw data<\/strong><\/h2>\n\n\n\n
![\"data](\"https:\/\/data36.com\/wp-content\/uploads\/2016\/11\/data-collection-sample-log-1024x357.png\")
What kind of raw data should you collect?<\/strong><\/h2>\n\n\n\n
What kind of data should you not collect?<\/strong><\/h2>\n\n\n\n
Conclusion<\/strong><\/h2>\n\n\n\n
\n
Tomi Mester<\/em><\/strong><\/p>\n\n\n\n
Tomi Mester<\/em><\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"