1. Author number in each field in each year

   • Each author is someone who at least one paper in some field in a given year, identified by unique names on ArXiv (which is not perfect).

   • If an author publishes in multiple fields in a given year, they are counted as a fraction of an author in each field, so that their total contribution to the author count is 1. 

     • I measure authorship based solely on the union of all fields that an author publishes in within a given year. It is not dependent on the relative number of publications in different fields for a given author, or on the number of co-authors on different papers. This helps control for differences in publishing practices in different fields. 

   • The goal here is to extract a metric that reveals switching behavior. Authors who make sustained transitions between fields will show up as +1 author in the new field. 

2. Net transition count per year of authors who switch from one field to another

   • The net transition count from field A to field B is how many authors who were in field A switch to a new field B in a specific year, minus the count of authors who switched in the reverse direction during that year. 

     • I calculate this in a way that accounts for fractional changes in authorship (straddling multiple fields). See Appendix 1 for details.

   • If an author makes a sustained transition from field A to field B in year X, I measure a transition count of +1 from A to B in year X only.

     • However, if someone switches back and forth multiple times, I count multiple transitions. 

     • If someone switches gradually over multiple years, they will contribute a fractional transition during each of those years. 

   • I choose to report the net transition count (the count of transitions in the forward direction minus the count in the reverse direction), because it is the ultimate relevant quantity. 

     • This is a better metric than the actual count in one direction because, if an author switched from A to B every year, they would (on average) contribute 0 to the net transition count (and this is the "correct" interpretation, as they are really just someone who straddles both fields, but never decides to change their average behavior). However, if one reported just the transition count in one direction, it would look deceptively large because of this author. Measuring net transitions seems more appropriate.

3. Net transition rate

   • This is just the net transition count between two fields in a given year, divided by the size of the source field in that year (the author count in the source field). It is the net flow out or in, per person in the field. 

   • Intuitively this is meant to capture "probability I would switch if I were in this field", loosely speaking.

4. Cumulative net transition count

   • This is the sum of the historical net transition count per year, over all previous years. 

   • It measures approximately how many total people have made a specific transition, up to that time. (Note, however, that it uses the net rate.)

In this post, I measure author trends on the preprint server ArXiv (widely used by math, physics, computer science, etc). The data is from a Kaggle dataset of all 2.5 million published papers, including author names, and the categories of their subject field. For all data analysis, and plots, and to mess with the data yourself, see my Github.

On ArXiv, each paper is published in a certain subject category (or a few categories). This lets us track which fields a given author publishes in each year.

Metrics

• Author number in each field 

  • If a specific author publishes in "stat.ML", "astro-ph", and "cond-mat.quant-gas" in a given year, then I count them as 1/3 of an author in each field during that year. 

    • This is not dependent on paper number. If they publish 5 papers in stat.ML, and one paper that is cross listed in astro-ph and cond-mat.quant-gas, then I still count them as 1/3 in each. I take the union of all published categories.

  • To get the full author count in a field during a year, I sum all of the contributions from individual authors.  

• Total transition count

  • Intuitive description: this is the number of authors in a given year who publish in a new category for the first time in that year. 

  • I track each specific author over time, and where they publish. When they change their distribution, I count that as a transition for the year they changed. I essentially treat the transition as uniformly distributed over gains and losses. I'll describe what I mean by that now: 

    • Example configuration: 

      • Imagine that an author 

        • Gains 0.1 in A, and 0.2 in B

        • Loses 0.05 in C, 0.05 in D, and 0.2 in E (note these sum to 0)

      • If I treat this transition as a random process, there is a 0.1 probability this person transitions into A, and if they do so, there is a probability of 1/6 that they came out of C, 1/6 they came out of D, and 2/3 that they came out of E.

        • Thus I count a transition event of

          • C to A  of amplitude 0.1/6

          • D to A  of amplitude 0.1/6

          • E to A  of amplitude 0.1/2

          • C to B of amplitude 0.2/6

          • ...

      • Hopefully this is clear, and intuitively captures what it means to make a fractional transition. 

  • To get the total transition count, I just sum up the transition matrix between all sources and target fields for each author in a given year.

    • I actually report the net transition rate in a given year (subtract the reverse measured transition counts), so this can be negative sometimes. 

• Transition rate per year

  • This is just the total transition count between two fields in a given year, divided by the size of the source field (author count in the source field). 

  • Intuitively this is representing something like the "probability I would switch out (or in) if I were in this field".

• Cumulative total transition count

  • This is just the sum of the historical transition events between fields in each year. 

  • It should give a decent representation of the total number of people who have switched between fields, over time.

    • Note: take care interpreting this. If people switch between fields many times in a big cycle, the overall transition rates could look large, but not mean quite what you think.

How I combined published fields

Note: as part of the analysis, I collapse various categories on Arxiv, ('astro-ph.CO': 'Cosmology and Nongalactic Astrophysics', and 'astro-ph.EP': 'Earth and Planetary Astrophysics') into one category, like "astrophysics". This is a choice I made.

Fundamental assumption that switching = value

I am assuming that switching fields is a good indicator of value, but people may also switch for other reasons. 

• Research in a field may be harder or easier than expected.

• People may switch just to "follow the crowd" rather than as a carefully thought out plan. People may feel a pressure to label their work as "the hot new field" even if it's the same type of work they have always been doing. 

ArXiv isn't complete (not representative of all scientific fields)

This is only ArXiv publishing data. Ideally I would want to include more databases across a broader range of fields (ie. bioRxiv), and characterize the paper categories in a more general way.

Different fields have different publishing behaviors

I have tried to make the analysis robust to the publishing behaviors of different fields, but I may have missed something. I have tried to make it so that the moment an author switches (really switches) is measured as a transition of 1 author during that year. 

• It should not matter whether authors publish in small groups, or large groups (many co-authors).

• It should not matter whether authors publish once every 5 years or many times per year. 

  • There is one problem, which is I may be undercounting the author number in a field if everyone only publishes every 5 years (because I count the authors publishing in a given year only, so I would count roughly 1/5 the true author number lurking in the background). 

  • However, the relative transition counts (the transition rates) measured are normalized to the author number, and should be robust (to this choice of 1 year vs. 5 year binning). If you want a more realistic author number, you could change the binning to 5 year size.

Author names are not a good way to identify unique authors

I use each author name as a unique string to identify an author, but this isn't correct. Some people have the same name, and I will therefore count them as the same person. To avoid this to some degree, I don't count any authors with more than 100 publications. This should help, but will then mess up the total author number. In the end, it's a tradeoff, and would be better to use something like Orchid identifiers.

Using names, which people share, could give the illusion of people switching fields if a new person appears who is in a growing field, with the same name as an older person in another field. But the fact that I don't see transitions from astro to other places might mean it's fine.

I made specific choices on how to group scientific field designations

I made specific choices on which archive subfields to consider "AI fields" and which to classify together as quantum physics, for example. Changing this will change conclusions slightly. This is slightly more complicated than it first seems: for example, astrophysics had a change of it's ArXiv organizations at one point, so there's massive flux between fields within ArXiv as people adjusted.

I measure net author switching rates

One might argue that the absolute switching rates are a more relevant metric. For example, if the field of AI is growing at rate X, but what's actually happening is people are transitioning into AI with rate 3X and out with rate 2X, that's probably worth knowing. However, it's not obvious how to measure this well (since it depends on the timing bin size you choose to measure transitions on). The net rate is nicely independent of the timing bin size.