Repository navigation
Suggestion - Post library download statistics/analytics numbers & graphs #1078
Description
Does CDNJS keep track of how many times individual files are pulled from their CDN? It would be absolutely awesome if CDNJS did this and then had up-to-update charts and data tables showing how many downloads each file is getting.
This would be epic because it would allow developers to choose which library versions they want their users downloading. The developer would want to choose the library version that is obviously compatible with their code, but also they would choose the one that is downloaded the most over the past X amount of time.
For example, lets say the latest jQuery just gets released and put on CDNJS. A couple days pass and the stats for jQuery look like this for the past week:
jQuery 1.9.1 = 20,000 downloads
jQuery 1.9.0 = 50,000 downloads
jQuery 1.8.3 = 560,000 downloads
jQuery 1.8.2 = 120,000 downloads
etc...
The developer can look at this and know that it's more likely that their visitors are already going to have jQuery 1.8.3 cached as opposed to 1.9.1 since it's new. So as long as their code is 1.8.3 compatible, they would choose this one.
And since these numbers change over time, maybe a month later the developer comes back to CDNJS and see's now that the 1.9.1 stats are higher than 1.8.3, so again, as long as his code is 1.9.1 compilant, he could safely switch his site to use 1.9.1 since his visitors are now more likely to already have 1.9.1 cached.
Does this make sense? To me it would be EXTREMELY useful. The whole point of CDNJS is so that developers share libraries and resources. So over time, as more and more libraries get added to CDNJS and more and more versions of those libraries are added, it would be invaluable to have a tool like this in order for the developers to make informed decisions based on which libraries and resources are being shared the most.
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
Activity
ryankirkman commentedon Mar 22, 2013
@Jakobud Great suggestion Jake. You're absolutely right that this would really useful, and it is a popular request: #405
We're brainstorming solutions right now, so we're glad to have you as part of the conversation.
Lockyc commentedon Mar 23, 2013
Closed old issue #405 continue conversation here
thomasdavis commentedon Jun 27, 2013
Tagged as high priority, anyone have any brilliant ideas yet on how to parse a few billion lines?
Jakobud commentedon Jun 27, 2013
How many lines is the typical log file? Do you split the log files up to one-per-day or smaller? Do the log files simply say what http://path/file was downloaded? Or does it have references to database row id's (id's of each filename which I assume are stored in a database)?
ryankirkman commentedon Jun 28, 2013
Each edge location (currently 23) is treated independantly from every other.
So what we have is one or more log files per edge location per day, and we
are getting a significant number of hits.
On Friday, June 28, 2013, Jake Wilson wrote:
Jakobud commentedon Jun 28, 2013
If you could post excerpts of the log files, that would be a place to start.
Jakobud commentedon Jul 31, 2013
Any progress on this? You guys need any help with it? I know there are probably a lot of huge log files, but I think it would only be a matter of a simply python script that streamed in the log files and saved out the data to a database or something like that. It would be a long running process but it probably wouldn't be that complicated really.
Jakobud commentedon Aug 7, 2014
FYI, I don't know if cdnjs utilizes AWS services on the backend or not, but this is an interesting article that is potentially very relevant to this issue:
http://aws.amazon.com/blogs/aws/all-your-data-fluentd/
It discusses using software called Fluentd to stream logfile changes into data storage. So for CDNJS, it could stream library access logs into some sort of usage database that could be used to display usage statistics.
Jakobud commentedon Aug 7, 2014
Also, FYI you guys could get someone to help you with a solution for this if you could divulge details about your logging. How it works, where the files are stored, give us access to a day or weeks worth of logs, etc... Someone could figure out a solution for you.
Jakobud commentedon Aug 7, 2014
Another suggestion for you guys, just make your logs public. Put them up on AWS S3 or something and allow anyone to grab them. I GUARANTEE someone (or multiple people probably) will come up with an analytics solution for you.
Jakobud commentedon Dec 11, 2014
Just wanted to reach out regarding this issue again. I'll say it again, provide some example log files and someone somewhere will put together a parser for you that will pull library download stats.
PeterDaveHello commentedon Dec 11, 2014
ping @thomasdavis
IonicaBizau commentedon May 23, 2015
Creating an api service for cdnjs would be nice. Something like:
Then, we can use this service to fetch the stats in the cdnjs website. 🍀
PeterDaveHello commentedon May 24, 2015
Stats from website is easy, but people want the stats from cdn, I remember that cloudflare didn't give us that info or access log.
cc @thomasdavis @ryankirkman @terinjokes
ryankirkman commentedon May 24, 2015
We can get access to the logs, but the log volume is so large we need to
figure out an aggregation strategy
On Sun, May 24, 2015 at 12:28 AM Peter Dave Hello [email protected]
wrote:
18 remaining items
dazbradbury commentedon Apr 8, 2019
Looks like this issue has been pretty stagnant - is there now an alternative / feasible solution for determining library usage stats or percentages?
Taking the jQuery example - as a site owner you care about % of users arriving with the required jquery version already cached, and any stats cdnjs can provide would be awesome in determining that.
MattIPv4 commentedon Apr 18, 2019
Currently waiting on Cloudflare to establish a way for us to have stats/log access for the
cdnjs.cloudflare.comdomain. Will post updates as I get them.MattIPv4 commentedon Apr 18, 2019
Noted from #6186 that more in-depth stats would be useful such as country breakdowns.
MattIPv4 commentedon Jun 6, 2019
@dknecht Please can we use this issue to track any updates on further stats/logs access to the
cdnjs.cloudflare.comdomain. Thanks :)