7 posts tagged with "insight"

We analyzed more than 5,000,000,000 rows of GitHub event data and got the results here. In this report, you'll get interesting findings about open source software on GitHub in 2022, including:

• Top languages in the open source world over the past four years
• Geographic distribution of developer behavior
• Developer behavior distribution on weekdays and weekends
• Popular open source topics
• The most popular repositories in 2022
• The most active repositories over the past four years
• Who gave the most stars in 2022
• The most active developers since 2011
• Appendix

Top languages in the open source world over the past four years​

This chart ranks programming languages yearly from 2019 to 2022 based on the ratio of new repositories using these languages to all new repositories.

Insights:

• Python surpassed Java and moved to #3 in 2021.
• TypeScript rose from #10 to #6, and SCSS rose from #39 to #19. The rise of SCSS shows that open source projects that value front-end expressiveness are gradually gaining popularity.
• The two languages Ruby and R dropped a lot in ranking over the years.

Rankings of back-end programming languages​

The programming languages used in a pull request reflect which languages developers used. To find out the most popular back-end programming languages, we queried the distribution of programming languages by new pull requests from 2019 to 2022 and took the top 10 for each year.

The chart data indicates:

• Python and Java rank #1 and #2 respectively. In 2021, Go overtook Ruby to rank #3 in 2021.
• Rust has been trending upward for several years, ranking #9 in 2022.

Geographic distribution of developer behavior​

We queried the number of various events that occurred throughout the world from January 1 to September 30, 2022 and identified the top 10 countries by the number of events triggered by developers in these countries. The chart displays the proportion of each event type by country or region.

The chart shows that:

• The events triggered in the top 10 countries account for about 23.27% of all GitHub events. However, the number of developers from these countries is only 10%.
• US developers are most likely to review code, with a PullRequestReviewEvent share of 6.15%.
• Korean developers prefer pushing directly to repositories (PushEvent).
• Japanese developers are most likely to submit code via pull requests, with a PullRequestEvent share of 10%.
• German developers like to open issues and comments, with IssueEvent and CommentEvent accounting for 4.18% and 12.66% respectively.
• Chinese developers like to star repositories, with 17.23% for WatchEvent and 2.7% for ForkEvent.

Notes:

• In 2022, 17,062,081 developers had behavioral events, and 2,923,523 of them have the Location field, so the sampling rate is 17.13%
• GitHub identifies 15 types of events. We only show commonly used types. Comment Event includes CommitCommentEvent, IssueCommentEvent, and PullRequestReviewCommentEvent. Others includes MemberEvent, CreateEvent, ReleaseEvent, GollumEvent, and PublicEvent.

Developer behavior distribution on weekdays and weekends​

We queried the distribution of each event type over the seven days of the week.

Insights:

• Developers are most active on weekdays, with 77.73% of events occurring on weekdays.

The distribution of specific events​

Insights:

• Pull Request Event, Pull Request Review Event, and Issues Event all have the highest percentage on Tuesdays, while the lowest percentage is on the weekends.
• The amount of Push Event, Watch Event, and Fork Event activities are similar on weekdays and weekends, while the Pull Request Review Event is the most different. Watch Event and Fork Event are more personal behaviors, Pull Request Review Events are more work behaviors, and Push Events are used more in personal projects.

Popular open source topics​

Each year, technology introduces new buzz words. Can we gain insight into technical trends through the open source repositories behind the hot words? We investigated five technical areas: Low Code, Web3, GitHub Actions, Database, and AI.

Activity levels of popular topics​

We queried the number of open source repositories associated with each technical area, as well as the percentage of active repositories in 2022.

This figure shows that open source repositories in the Low Code topic are the most active, with 76.3% being active in 2022, followed by Web3 with 63.85%.

Popular topics over the years​

We queried the following items for each technical area from 2015 to 2022:

• The annual increment of repositories
• The annual increment of collaborative events
• The number of developers participating in collaborative events
• The annual increment of stars

Then, we calculated the growth rate for each year which can reflect new entrants, developer engagement in this technical field, and the industry's interest in this area. For 2022, we compare its first nine months with the first nine months of 2021.

We can see that 2020 is the peak period of project development, with a 313.43% increase in new repositories and a 157.06% increase in developer collaborative events. The industry's interest increased most significantly in 2021, reaching 184.82%. In 2022, the year-on-year growth data shows that the number of new repositories decreased (-26.21%), but developer engagement and industry interest are still rising.

* Time range of 2022: 01.01-09.30, excluding bot events and forking repositories

Whether it is the creation of new repositories, developers, or the interest of the industry, the Web3 ecosystem has grown rapidly in recent years, and the growth rate of new repositories peaked at 322.65% in 2021.

* Time range of 2022: 01.01-09.30, excluding bot events and forking repositories

The annual increase of GitHub Actions repositories has been declining, but developer engagement and the industry's interest are still increasing slightly.

* Time range of 2022: 01.01-09.30, excluding bot events and forking repositories

As an infrastructure project, the Database project's threshold is high. Compared with projects in other fields, a database project has a stable growth rate.

* Time range of 2022: 01.01-09.30, excluding bot events and forking repositories

After two years of high growth in 2016 and 2017, open source projects in AI have been growing gradually slowly.

* Time range of 2022: 01.01-09.30, excluding bot events and forking repositories

The most popular repositories in 2022​

The number of stars is the most visible indication of the popularity of open source projects. We looked at the 50 projects that received the most stars from January 1 to September 30, 2022. We found that:

* Time range: 2022.01.01-2022.09.30, excluding bot events

The most active repositories over the past four years​

Here we looked up the top 20 active repositories per year from 2019 to 2022 and counted the total number of listings per repository. The activity of the repository is ranked according to the number of developers participating in collaborative events.

Insights:

• Microsoft has the most repositories on the list, with five.

• tensorflow/tensorflow and kubernetes/kubernetes both dropped out of the top 20 after three consecutive years on the list (2019 to 2021).

• New to the 2022 list are archway-network/testnets, element-fi/elf-council-frontend, solana-labs/token-list, education/GitHubGraduation-2022, taozhiyu/TyProAction, NixOS/nixpkgs, rust-lang/rust.

• Time range: 2022.01.01-2022.09.30, excluding bot events

Who gave the most stars in 2022​

We queried the developers who gave the most stars in 2022, took the top 20, and filtered out accounts of suspected bots. If a developer's number of star events divided by the number of starred repositories is equal to or greater than 2, we suspect this user to be a bot.

We found that until September 30, 2022, the developer who starred the most repositories had starred a total of 37,228 repositories, an average of 136 repositories per day.

* Time range: 2022.01.01-2022.09.30, excluding bot events

The most active developers since 2011​

We queried the top 20 most active developers per year since 2011. This time we didn't filter out bot events.

We found that the percentage of bots is becoming larger and larger. Bots started to overtake humans in 2013 and have reached over 95% in 2022.

Appendix​

Term description​

• GitHub events: GitHub events are triggered by user actions, like starring a repository or pushing code.
• Time range: In this report, the data collection range of 2022 is from January 1, 2022 to September 30, 2022. When comparing data of 2022 with another year, we use year-on-year analysis.
• Bot events: Bot-triggered events account for a growing percentage of GitHub events. However, these events are not the focus of this report. We filtered out most of the bot-initiated events by matching regular expressions.

How we classify technical fields by topics​

We do exact matching and fuzzy matching based on the repository topic. Exact matching means that the repository topics have a topic that exactly matches the word, and fuzzy matching means that the repository topics have a topic that contains the word.

The OSS Insight website displays the data changes of GitHub events in real time. GitHub events are activities triggered by user actions on GitHub, for example, commenting and forking a repository. In nearly seven weeks, GitHub events increased by about 150 million, from 4.7 billion to 4.85 billion. GitHub events are booming!

This post dives deeply into GitHub event trending, why GitHub events are surging, and whether GitHub's architecture can handle the increasing load.

Historical data analysis​

The OSS Insight database includes all the GitHub events since 2011. When we plot the number of events by year, we can see that since 2018 they have been increasing rapidly.

The figure below shows how long it takes to grow each billion events in GitHub.

It's taking less and less for GitHub to generate 1 billion events. It took more than 6 years for the first billion events and only 13 months for the last billion!

The secret behind the exponential growth of GitHub events​

GitHub Actions was released in October 2018. Since August 2019, it has supported continuous integration and continuous delivery (CI/CD), and it has been free for open source projects. Therefore, projects hosted on GitHub can automate their own development workflows, and a large number of automation-related bot applications have appeared on GitHub Marketplace. Could GitHub events' data growth be related to these?

To find the answer, we divided the events into data from humans and data from bots and plotted them with the following histogram. The blue columns represent the human data, and the yellow columns represent the bot data.

As you can see, the proportion of GitHub bot events has increased each year. In 2015, they were only 1.23% of all events. In early July of this year, they reached 13.2%. To show the data changes of bot events more clearly, we made the following line chart.

This figure shows that since 2019, bot events have been grown faster than before. As Mini256, a TiDB community contributor said in Love, Code, and Robot — Explore robots in the world of code:

For now, rough statistics find that there are more than 95,620 bots on GitHub. The number doesn't seem like so much, but wait...

These 95 thousand bot accounts generated 603 million events. These events account for 12.82% of all public events on GitHub, and these GitHub robots have served over 18 million open source repositories.

Bots are playing an increasingly important role on GitHub. Many projects are handing over automated work to bots. We expect that GitHub events will grow faster in the future.

When will GitHub reach 10 billion events?​

How many GitHub events will there be by the end of 2022? We fit predictions to GitHub historical data.

It's estimated that by the end of 2022, GitHub events will reach 5.36 billion.

According to this prediction, GitHub events will exceed 10 billion in February 2025.

Can MySQL sharding support such a huge amount of data?​

GitHub uses MySQL as the main storage for all non-git warehouse data. The rapid growth of data volume poses a great challenge to GitHub's high availability. In March 2022, GitHub had 3 service disruptions, each lasting 2-5 hours. The official investigation report shows the MySQL database caused the outages. During peak load periods, the GitHub mysql1 database (the main database cluster in GitHub) load increased. Therefore, database access reached the maximum number of connections. This affected the performance of many GitHub services and features.

In fact, over the past few years GitHub has optimized its databases. For example, it added clusters to support platform growth and partitioned the main database. But these improvements did not fundamentally solve the problem. In the near future, GitHub events will exceed 5 billion, or even 10 billion. Can MySQL sharding support such data surge?

Data sources​

All the analysis data in this article comes from OSS Insight, a tool based on TiDB to analyze and gain insights into GitHub events data.

You can use it to easily get insights about developers and repositories based on billions of GitHub events. You can also get the latest and historical rankings and trends in technical fields.

In this chapter, we will share with you some of the top JavaScript Framework repos(JSF repos) on GitHub in 2021 measured by different metrics including the number of stars, PRs, contributors, countries, regions and so on.

Note:

1. You can move your cursor onto any of the repository bars/lines on the chart and get the exact number.
2. The SQL commands above each chart are what we use on our TiDB Cloud to get the analytical results. Try those SQL commands by yourselves on TiDB Cloud with this 10-minute tutorial.

Star history of top JavaScript Framework repos since 2011​

The number of stars is often thought of as a measure of whether a GitHub repository is popular or not. We sort all JavaScript framework repositories from GitHub by the total number of historical stars since 2011. For visualizing the results more intuitively, we show the top 10 open source databases by using an interactive line chart.

In this chapter, we will share with you some of the top low-code development tools repos (LCDT repos) on GitHub in 2021 measured by different metrics including the number of stars, PRs, contributors, countries, regions and so on.

On this page, we will share with you many deep insights into open source databases, such as the database popularity, database contributors, coding vitality, community feedback and so on.

We’ll also share the SQL commands that generate all these analytical results above each chart, so you can use them on your own on TiDB Cloud following this 10-minute tutorial.

In this chapter, we will share with you some of the top programming language repos (PL repos) on GitHub in 2021 measured by different metrics including the number of stars, PRs, contributors, countries, regions and so on.

In this chapter, we will share with you some of the top Web Framework repos (WF repos) on GitHub in 2021 measured by different metrics including the number of stars, PRs, contributors, countries, regions and so on.