Abstract

Academic freedom is a key input to academia and the production of new ideas. Current incentives in academia have resulted in a publish-or-perish culture, the replication crisis, and undervalued, overlooked researchers and research. We propose a novel peer-to-peer protocol for scientific collaboration using an improved research impact metric, as well as market-based approaches of prediction markets, replication markets, and information prizes. We aim to address the shortcomings of the current peer review system by incentivising the production of quality knowledge that creates signals for truth, impact, replicability, and novelty.

1 Introduction

There are many problems in today’s academic and research systems. Among these is the replication crisis, which exists despite the introduction of modern peer review in the 1970s.

The latter was intended to ensure that only high quality research would be published and has not only comeshort of that aim, but has also stifled innovation and creativity by subjecting research proposals to constraints of efficient use of finances, resulting in the mostly incremental improvements of safe research, leaving little room for the non-linear nature of breakthrough academic creativityto flourish [1].

Other problems include high levels of competition resulting in a publish-or-perish culture, which impedes the type of long-term thinking necessary for research breakthroughs, as well as other malignancies, such as the overwork of graduate students, p-hacking, dwindling time forprofessors to conduct research, and others. In light of the rise of distributed ledger technologies (DLTs), there has re-arisen an interest insolving these problems using distributed or decentralised protocols. In this paper, we propose a novel approach to addressing some of the issues apparent in the academic system using a cryptoeconomic network. The key breakthrough is unlike the system today that is optimised towards prestige we incentivise the creation of quality, truthful knowledge that is measured on it's merit alone.

This new and permissionless protocol that has the following characteristics:

1. Academic freedom: Academic freedom faces an increasing number of obstacles, such as more competition for tenure-track positions and funding, as well as a publish-or-perish culture that hinders exploratory research. We propose re-introducing some of the scientific norms thatwere in place prior to the introduction of modern peer review in the 1970s, as well as newmodes of collaboration and communication that will carry us into the future.
2. Pursuit of truth: By incentivizing the discovery and promotion of novel and overlookedresearch, we aim to bring new and valuable ideas the attention they deserve much faster thanthe current peer-review system.
3. Incentivizing submission of all research outputs: All forms of research should be publishable. This includes negative results, which are often not published, despite being valuable.In addition to making better use of already-exhausted resources, publishing negative resultscan also help other researchers avoid unnecessary approaches in their own research, thusincreasing the output of the network. Additionally, any research item that contributes to collective knowledge, including graphs, notebooks, stand-alone experiments, individual theoremsand proofs, etc. can be published and recognized for its value.
4. Signals for Impact and Truth: We aim to supplement peer review with prediction markets,which incentivizes more objective assessments of the quality and truthfulness of research, sinceparticipants in the market have a financial stake in the outcome of the research.
5. Knowledge should be preserved: The network is built on Arweave, which allows for thepersistent storage of information that cannot be taken offline, as well as enabling members ofthe network to contribute to its storage themselves. Publishing is completely permissionless- research can be published by anyone, anywhere, with little to no barriers to entry.
6. Funding Freedom: There is a large disconnect between those who choose what research isfunded with public money, and the taxpayers who supply that money. Redirecting the leversof decision-making from the former to the latter enhances democracy, as well as increasestrust in the results of the research. This fact is especially pertinent in light of the growingdistrust in academic and scientific institutions.
7. Weighted contributions: While the question of how much authors contributed to a paperis a complicated one, we propose an alternative to the traditional ordinal approach to listingauthors: weighted contributions. While this approach cannot address the multi-dimensionalnature of contributions in a number of different fields, it allows for more flexibility than thetraditional system.
8. Universality: Science should be universal, with no boundaries or borders. We introducea new approach to the funding, publication, and rewarding of research that enables widerparticipation in the scientific method for everyone, everywhere.

2  Rosetta Network Structure

The structure of the network is as follows:

1. Researchers `r_1,r_2....,r_abs(R) in R.`
2. Research items `t_1,t_2....,t_abs(T) in T.`
   Each researcher has an associated set of research items which they have (co-)authored,and each research item has an associated set of researcher(s) who (co-)authored it. Suppose that researcher r has a set of n research items. Then we denote r’s research items as `r^1,r^2,....r^n`, and refer to the collection of research items as `r^**`. Likewise, suppose that re-search item t has m authors. Then we denote t’s authors as `t^1,t^2,....t^m` and refer to all the authors of t as `t^**`.
3. Denote the percent contribution of `r_i` to `t_j` as `c_i^j`. Note that `sum underset(i in t_j^**)``c_i^j=1.`When a research item is submitted, each researcher will be assigned their own individualcontribution score. The value of each researcher’s contribution will be decided by the themindividually.<br>The researchers may disagree with each other about individual contributions. If so, conflictresolution will be handled as follows:
   (a) The Rosetta network will request the researchers to work out their difference and cometo an agreement.
   (b) If an agreement is not reached, then if a strict majority of the researchers agree uponthe distribution of contribution scores, those will be the final contribution scores.
   (c) Otherwise, each researcher will be assigned a contribution scored based upon the me-dian/mean desired contribution scores.
4. Denote the set of research items that `r_i` cites as `b_i`, and the set of research items that cite `r_i` as  `b_i`. Suppose that `r_i` cites n research items. Then we denote these as `b_{i1},b_{i2},....b_{i n}`. Likewise,suppose that n research items cite `r_i`. We denote these as `b^{i1},b^{i2},....b^{i n}`.
5. Citation matrix W, which also describes a DAG. Let `w_{ij}` denote the weight that paper i givesto research item j; note that `sum underset(j in R)``w_{ij}=1, AA_i in T`. If not specified by the researchers, thedefault is `w_{ij}=1/abs(b_i) AA_j in b_i`.
   Note that highly original work differs from, for example, survey papers in that the formeroften does not rely on much previous work. Thus, we may want to relax the constraint thatthe sum of the weights equals 1, in order to account for how much the research item actuallyrelies on other research items. If this is done, one problem it opens up is how to incentivisehonest descriptions of reliance on prior work.
6. Each research item `t_j` has a set of corresponding tokens `o_j` called knowledge tokens. Thesetokens are divisible, and fungible with other tokens corresponding to the same research item,but are not fungible with tokens from other research items (although they may be traded forthem).
7. There is a network-wide currency which we denote with the symbol $.

3 Rosetta Economy

Rosetta tokens are minted every block according to some schedule. For every block `\i`, we denotethe tokens minted in that block as `\L_i`. These tokens are distributed to knowledge tokens holdersproportionally to their research items’ allocation of the block reward via an allocation rule, afunction which we denote as `\a`.

3.1 Allocation Rule

An allocation rule maps `\W` to a reward vector `upsilon in RR^abs(T)`; that is, `a:W mapsto upsilon in RR^abs(T)`. As a basicstarting point, we will use the PageRank algorithm as the allocation rule. PageRank values are normalized to sum to 1, and the allocation rule follows immediately. Additionally, PageRank alsoaccommodates weighted edges, which are optional for researchers to use in the citation graph.<br>For some block `\i` let `upsilon_i` be the vector corresponding to the PageRanks induced by the citationmatrix in that block’s timeframe, and let `upsilon_i^j` be the j^th entry in `upsilon_i` (corresponding to research item `t_j`). Then the amount allocated to research item `t_j` is `upsilon_i^j L_i`.This amount is then distributed proportionally to holders of `o_j`.

3.1.1 Attack Vectors

As demonstrated in [8], PageRank is resistant to a number of attack vectors. We will briefly describe the findings of [8], as well as extra considerations that must be accounted for and future work to make the allocation rule as fair as possible.

The authors considered a PageRank-index for assessing the contributions of academics,  and compared it to the h-index. In order to account for the varying contributions of authors to their papers, they defined a metric which gave higher weight to the first researchers listed on a paper, although noted that fields where this is not the convention could be accounted for, thus allowing for the use of the earlier-authors-higher-weight metric without loss of generality.

The authors applied PageRank index to three simulations, each with a different “attack vector”(though the behavior could arise from legitimate intentions), and to papers from two fields: quantum game theory, and high energy physics between January 1993 and April 2003.The three scenarios were

1. Scenario 1: The impact of self-citing low impact publications. In this scenario, there are two groups of authors: a group which excessively self-cites with low-impact (low citation count) papers, and a group which does not.
2. Scenario 2: The impact on ‘singleton’ authors. In this scenario, there are two groups of authors: a group of authors who collaborate often, and another where the authors have substantially fewer authors on average compared to the first group.
3. Scenario 3: Handling the balance between quality and quantity of publications. In this scenario, there are two groups of authors: a group which publishes many low-impact papers, and another where the authors publish fewer papers of higher quality

In all three scenarios, the PageRank-index was not nearly as affected by the types of authors compared to the h-index, and PageRank only slightly favored groups which had higher impacts asmeasure by h-index.

Experiments conducted with papers from quantum game theory and high energy physics showed substantial differences between impacts as measured by h-index and PageRank-index, with the latter having several advantages, such as identifying undervalued authors and giving them higher rankings.

The authors reference [9] in their conclusion in relation to mitigating potential attack vectors.However, the attacks vectors described there are mostly based on cyclic citations, which normally donot occur in academic networks (though they can be accounted for, both illegitimate and legitimate, with the latter potentially arising in the case of living, breathing documents). Nevertheless, the heuristics described in [9] can be implemented, and the success of PageRank and the modifications that have been made by Google since PageRank’s inception offer evidence that the allocation rule can beadjusted to account for malicious activity.

It is also important to note that since researchers are non-anonymous, any malicious behaviorcan be noticed easily by the network, and it is possible to adjust the allocation rule to exactconsequences for those researchers.

3.1.2 Other Considerations

The authors of [8] point out that different fields have different citation distributions, which influences the rankings of researchers in those fields relative to the other fields. They suggest the possibility ofhaving separate indices for different fields. If this proposal is implemented in Rosetta, there would be a need to determine the distribution of Rosetta tokens to those different fields.

Note that robustness to the attack vectors means that researchers are not incentivised to publish or-perish, satisfying one of the desiderata of Rosetta.

It is also important to consider the possibility of coalition formation among researchers. This would need to be accounted for, and there may already exist viable approaches from the fields of algorithmic game theory and cooperative game theory, drawing on insights from resource-based games, strategic coalition formation, and others.

3.2 Knowledge Tokens

Each research item comes with associated knowledge tokens. These knowledge tokens have multiple uses; as described above, they determine the distribution of block rewards to token holders. However, they also have several more uses. Here we explore their uses for prediction and replicationmarkets.

3.2.1 Prediction Markets

‍Prediction markets are markets where participants can place bets on the outcomes of events. Theyhave been proposed as a method to assess the value of scientific research, as well as the likelihoodof a study being replicated [7] [4] [2] [3] [6]. In the context of Rosetta, knowledge tokens facilitateprediction markets. Participants in the market can buy knowledge tokens in accordance withtheir beliefs about the truthfulness and impact of the research. In equilibrium, the price of theknowledge tokens should equal the aggregate belief about the future rewards allocated to thoseknowledge tokens, which is a function of PageRank-index, which we have argued is a valuable toolfor assessing impact in citation networks.

3.2.2 Replication Markets

Furthermore, knowledge tokens can also be used to facilitate a version of replication markets.Suppose that a percentage of the rewards corresponding to a particular research item is availablefor the purpose of rewarding replications of the research. If the value of the rewards is high, thenincentives are high to attempt a replication. Thus, the research which is being relied upon the mostto be correct by the network (as determined by the PageRank-index of the corresponding researchitem) is the research which has the highest rewards in an attempted replication or triangulation(the latter attempts to assess the validity of claims in the original research from a different angle).

Note the underlying exchange: the replicators are awarded with a portion of the rewards of theoriginal authors, and the original authors are rewarded by being cited by the replicators. There aremany potential ways to augment this exchange - for example, the original authors and replicators,if they are on good terms and agree with each other about the states of the original research andattempted replication, could exchange some portions of each others’ knowledge tokens/rewardscorresponding to those knowledge tokens.

However, if the original authors and replicators do not come to an agreement, then the matterwill be taken to “trial”, where the jury is the Rosetta network itself, or a subset of it, particularlyresearchers from within the field. Voting power in this jury system can be based on the PageRankindex, and participants can delegate their voting power to individuals that they trust if they do notwant to participate in the voting itself. Matters to take into consideration include how accurate theoriginal research was, how well the replication was performed, and whether the replication providedvaluable results or insights.

Notice that this system incentivizes the original authors of papers to make their results as easilyreplicable as possible - for example, by pre-registering their hypotheses and methods, for fieldswhere doing so is applicable. Furthermore, it also incentivizes purchasers of knowledge tokens toconsider the replicability of the pertinent research items, since the amount awarded to replicatorsvaries depending on how accurate the original research was.

The process would be the same in the event of additional replication/triangulation attempts afterthe first one. The original authors and replicators could agree on some distribution of rewards, orthe matter can go to “trial”.

3.2.3 Information Prizes

One of the problems in the current academic system is that researchers who want to work onresearch that is truly transformative but unlikely to receive funding have no recourse. However,if they can sell some portion of their knowledge tokens in advance, they may find funders. Sincethere is an incentive to buy the knowledge tokens corresponding to research items which have ahigh likelihood of being impactful, the economic incentives are aligned in favor of such researchers,in comparison to the current grant system.

Overall, what is likely to emerge is a type of scientific portfolio manager. Those who are genuinely talented at assessing existing or newly-published research, and/or assessing researchers andtheir proposals, will become successful in these prediction and replication markets, while simultaneously contributing to the production of impactful knowledge.

4 Gordian Knot Market

4.1 Sinks

There are a number of potential sinks for this network. The most obvious are the fees that researchers often have to pay in order to get their work published. In Rosetta, publishing a researchitem would be based on the cost of publishing data permanently on Arweave, which as of thiswriting is approximately 3 USD/GB (although there may be good reasons to increase this for thepurposes of the network). Since publishing on Arweave is much more affordable than publishing ina traditional journal, some portion of the difference in price can be directed towards the network.

Another major potential source of revenue is the cost savings that come from higher rates ofreliability and replicability of research. If the network can achieve these aims, then the potentialbenefits are massive. One study put the amount of money lost in the life sciences/biomedical field in the United States alone at 28 billion USD per year [5]. It is safe to assume that the number ishigher when accounting for research outside of life sciences, and outside of the United States.

Additionally, another potential source could be the increased throughput in research resultingfrom the ability to publish any research item at any time (including negative results), helps improvethe productivity of other scientists by also allowing them to rely on research more quickly then theywould be able to otherwise. Like the savings from replication, it is not entirely clear how to actuallyfunnel the money into the network. However, the value proposition is clear, and if the network canachieve these aims, the question of funding reduces to convincing the relevant parties that thissystem is superior (in some ways, at least) to the legacy system.

4.2 Commercialization

It is possible that some entity (potentially including some members of the network) commercialisesand profits from the research produced by the network. Directing some of the profits towards thenetwork may contribute to its sustainability and attractiveness. As of this writing, there are manyscience-based DAOs either already in existence, or currently being created, which can use Rosettaboth as a platform for publishing their research, and as a way to receive rewards as they try toachieve their goals.

5 Conclusion

We introduced Rosetta, a decentralized research network which aims to solve the problems of thecurrent academic system: in particular, the publish-or-perish culture, the replication crisis, andundervalued researchers.

References

1. Donald W Braben. Scientific Freedom: The Elixir of Civilization. John Wiley & Sons, 2008.
2. Colin F Camerer et al. “Evaluating replicability of laboratory experiments in economics”. In:Science 351.6280 (2016), pp. 1433–1436.
3. Colin F Camerer et al. “Evaluating the replicability of social science experiments in Natureand Science between 2010 and 2015”. In: Nature Human Behaviour 2.9 (2018), pp. 637–644.
4. Anna Dreber et al. “Using prediction markets to estimate the reproducibility of scientificresearch”. In: Proceedings of the National Academy of Sciences 112.50 (2015), pp. 15343–15347.
5. Leonard P Freedman, Iain M Cockburn, and Timothy S Simcoe. “The economics of reproducibility in preclinical research”. In: PLoS biology 13.6 (2015), e1002165.
6. Michael Gordon et al. “Are replication rates the same across academic fields? Communityforecasts from the DARPA SCORE programme”. In: Royal Society open science (2020).
7. Robin Hanson. “Could gambling save science? Encouraging an honest consensus”. In: (1995).
8. Upul Senanayake, Mahendra Piraveenan, and Albert Zomaya. “The pagerank-index: Goingbeyond citation counts in quantifying scientific impact of researchers”. In: PloS one 10.8 (2015),e0134794.
9. Hui Zhang, Ashish Goel, Ramesh Govindan, et al. “Making Eigenvector-Based ReputationSystems Robust to Collusion”. In: (2004).