Skip to content

[Proposal] New Top Language Detection Algorithm  #1600

Closed

Description

@kitswas

Is your feature request related to a problem? Please describe.

This is related to an earlier issue.
I am not quite happy with the way Top Languages are calculated (just counting the bytes).

Describe the solution you'd like

The new method is inspired by the h-index.

For a language L,
let x = byte count(currently in use) and y = number of repositories using L.

x*y = k^2
k = sqrt(xy)

Now we can use this k-metric to find the top languages and their weightage.
The higher the value of k, the more used is the language.

While this method is an improvement over both byte count and the number of repositories, it is in no way a magic wand.

Describe alternatives you've considered

None at the moment.

Additional context

I have gone through this issue mentioned in the FAQ.

P.S.: This is the first time I am trying to contribute to open source. Please point out if I make a mistake.

Activity

  1. anuraghazra commented on Feb 19, 2022

    @anuraghazra
    Owner

    Hi @kitswas thanks for the proposal, quite interesting.

    What do you think of the downsides of this approach? Can you mention any examples?
    And also how exactly this method is better than just counting bytes? Will be better to have a comparison.

  2. added
    ranksFeature, Bug fix, improvement related to ranking system.
    lang-cardIssues related to the language card.
    on Feb 19, 2022
  3. kitswas commented on Feb 19, 2022

    @kitswas
    ContributorAuthor

    What do you think are the downsides of this approach?

    The size of a program (sourcecode) to perform the same task varies when written in different languages. This metric does not take it into consideration.
    The alternatives (counting bytes and number of repositories with the language) are no better though.

    And also how exactly this method is better than just counting bytes?

    When a language appears in multiple repositories, it means that the developer has some degree of reliance on it. This is an attempt to factor it into the calculation.

    The square root in the new metric is used to tone down the percentage differences between languages.
    A general form of the index would be

    k = x^p * y^q
    

    where p and q are constants which control the influence of x and y on the metric, respectively.

    Will be better to have a comparison.

    Here's a theoretical comparison.
    Comparsion.pdf
    Comparison.png
    This developer has three repositories containing source-code for three websites. He used HTML for all his websites but JavaScript for only one.
    Yet, due to the massive size of the JS code, his most used language (according to the old metric) is JavaScript. While there's no doubt he has written a lot of code using JS, HTML is what he relied the most upon.

    For a real world comparison, I need data.

  4. kitswas commented on May 3, 2022

    @kitswas
    ContributorAuthor

    Here's how it would affect my current card.
    comparison table
    I admit, I am quite unhappy with Python hogging everything. I cannot use the exclude_repo option as it removes Python entirely.

    Note: Since I did not have the number of bytes, I used the percentages for calculation. The result will remain unaffected since we are finding a ratio. Also note that the final percentages are slightly higher than they should be as the orginal percentages for the selected languages add up to 99.36% only.

  5. rickstaa commented on May 7, 2022

    @rickstaa
    Collaborator

    Just for reference, in case people want to chip. Part of the discussion relating to this issue can be found in #1732.

  6. added
    proposalProposals to change code or documentation fundamentals.
    on May 8, 2022
  7. changed the title [-]New Top Language Detection Algorithm [/-] [+][Proposal] New Top Language Detection Algorithm [/+] on May 8, 2022
  8. removed
    ranksFeature, Bug fix, improvement related to ranking system.
    on Nov 18, 2022
  9. rickstaa commented on Sep 11, 2023

    @rickstaa
    Collaborator

    This was implemented in #1732.

  10. added a commit that references this issue on Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.lang-cardIssues related to the language card.proposalProposals to change code or documentation fundamentals.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions