Evaluating ideas for eLife’s Innovation Initiative

By Giuliano Maciocci and Naomi Penfold, PhD

eLife’s Innovation Initiative was set up to identify and support innovative, small-to-medium-scale open source projects aiming to improve the sharing and reuse of scientific discoveries. The open source aspect is important to us, as we believe it encourages reusability of the technologies we sponsor and provides the benefits of community development resources and feedback that can often make a difference in the success of early-stage technology projects.

Typically, we look for early-stage projects where we can add the most value by providing the funding, mentorship and exposure necessary to get them to a ‘Version 1’, the minimum viable product (MVP) we deem necessary to demonstrate the concept, showcase end-user functionality, and hopefully entice further funding and get on a path to long-term sustainability.

The open-source-for-open-access (or open scholarship) space we operate in is rich with creative individuals and teams overflowing with ideas. In order to evaluate which projects we should actively support, we have developed a set of internal evaluation criteria that provide us with a baseline for determining whether a project is a good fit to eLife and its mission, and whether the team or individual associated with the project has a solid understanding of their idea’s implications.

If you happen to be that individual or a part of that team, we welcome approaches about new project ideas by email to innovation at elifesciences dot org. Our Innovation Officer will work with you to understand the idea and, if there is an opportunity to support your project, they will invite a proposal. We encourage you to consider the following questions.

Q1. What is the project?

“If you can’t explain it simply, you don’t understand it well enough.”  

We expect to see a good understanding of the core idea of the project, demonstrated by a concise, jargonless explanation of the idea’s core benefits, and a well-defined understanding of its intended audience or beneficiary. We don’t need you to be “the X of Y”, or to “disrupt the [blank]”. We just need you to tell us what existing problem you’ll be solving in a way that demonstrates you truly understand it yourself.              

Q2. What, and whose, problem is being addressed?  

“Wouldn’t it be nice if” is generally not a great problem definition"  

Calling back to Q1, this is when we’ll expect to get into the real detail of an idea’s problem space, and that includes the problem’s primary audience.
“Wouldn’t it be nice if” is generally not a great problem definition. On the contrary, while many ideas would be nice in theory, we prefer to focus on those that directly address a clearly identified challenge experienced by a clearly identified group, or groups, of people.

Be prepared to provide context, including which group of actual humans will benefit from your idea, and evidence that the problem you plan to address reflects a real-world problem experienced by the group you have identified. While we greatly value personal experience, we also want to ensure that a problem space’s audience consists of a sample greater than one.

      Q3. Who is in the team?            

"If ideas are superheroes, we’re all about origin stories."

When it comes to technology, a great idea’s impact is only as good as its execution, and we want to make sure the people working on the project (together with our eventual support) have the resources, time and expertise available to get their idea off the ground with a sufficient level of quality – or have a plan as to how to obtain them. This means we welcome an honest evaluation of where you have the skills required to deliver the product and where your gaps are.

Just as importantly, we’re looking for passion and commitment, two qualities without which we are unlikely to engage. If you’re not passionate about your project, should we be?

We’ll also be looking to understand your background in order to contextualize your idea, and since innovation does not happen in a vacuum, we like to get a sense of what inspired you to solve a specific problem. In our experience, some of the most promising innovation projects are those that address problems their creators are intimately familiar with.            

Q4. What’s your ethos?            

"A good open source technology project is like a good research artifact: it’s crafted in such a way as to encourage reuse and derivative works."

eLife is a mission-driven non-profit, so understanding where your ideas stand from an ethical standpoint is important to us.

Specifically, we’re looking for ideals aligned with the principles we believe in: openness, transparency and the encouragement of reusability of technology assets for the greatest possible benefit to the greatest possible number of beneficiaries. We favor projects for which inclusivity and accessibility have been factored into development plans – important aspects which are, unfortunately, all too often treated as afterthoughts when planning an initial technology rollout.
All of this does not mean we expect you to be charity minded. We have no problem supporting project proposals that include a monetization strategy, particularly when this is necessary for sustainability later. However, we are cautious of projects where terms such as “proprietary”, “lock-in” and “intellectual property” may apply.
A good open source technology project is like a good scientific artifact: it’s crafted in such a way as to encourage reuse and derivative works. The ideal project proposal will include a reuse strategy, supported by a good understanding of best practices in open source, including distribution, documentation and licensing. Some idea of what type of project governance might be applied to the long-term management of community contributions is welcomed.              

Q5. What investment has the project had so far?          

Since we want to make sure that every collaboration we engage in adds maximum value, we like to understand what other funding has been made available, if any, and what commitments or conditions have been placed on the project as a result. We’re generally happy to co-fund a promising project, so long as we feel we can provide additional value and can ensure no impediments have been placed on the project’s ability to remain open and achieve its goals as we understand them.  

Q6. How far along is the project? Has it had any impact?          

The best proof of your commitment, understanding of the problem space and ability to execute can sometimes be found in work you have already accomplished towards your idea’s stated goals. This may be products, designs, evidence gathering, or more.

For the earliest-stage projects, we want to hear whether you understand what the first practical test of your idea could be, and what the results were if you’ve attempted this already. In other words, what could be the quickest (and often less polished) way to see if prospective users would even want to engage in the idea, and how could we measure that engagement?

Whether it’s a prototype, proof-of-concept or a live implementation with an already thriving community, the stage of an idea’s execution will directly influence how much impact our funding might have. It will also affect how much additional value we could provide through other contributions, such as helping to define requirements and provide design and development support, mentorship and project exposure.

An important consideration for us here is the balance between maturity, flexibility and risk. The earlier stage a project, the riskier, but early-stage projects also offer the most scope for us to work together with the team to help define its direction, governance and the scope of an eventual MVP. Conversely, more mature projects – particularly ones with an engaged community – have a better outlook for long-term sustainability and thus for the potential impact of our value-add or contributions.

Q7. Is it similar to anything else? How does it fit within the existing ecosystem? Lessons learned?          

OK, this is more like three questions, but they’re all related. While nice to have, novelty is not a necessary prerequisite for a successful collaboration under the eLife Innovation Initiative. But if you are improving on an existing solution, we want to make sure you’re intimately familiar with it and and can elaborate on what specific advantages your approach might have over the status quo.

A related, if perhaps less obvious, aspect of this is how the idea fits within the current ecosystem. Technologies seldom exist in a vacuum, and the infrastructure supporting the field of scholarly communication is made up of a vast network of interconnected technologies and standards with which any new technology proposal may need to coexist. It is therefore important that you demonstrate a good knowledge of the ecosystem within which your proposed technology will operate, and of the necessary intersections and dependencies with the other technology solutions in the ecosystem. You may not know this at the start of our interaction — we are happy to help connect you with people, organizations and information through which you can build this understanding.

Finally, we’re also looking for whether anyone has previously attempted to implement an idea, and what, if any, lessons have been learned from those attempts. We think past failures are highly valuable when used as learning experiences for future attempts at success.

Q8. What’s the project’s roadmap?          

Many of the project proposals we come across have only been thought out as far as their first release and sometimes not even that far. If, however, there is a roadmap, we like to understand what it is and how it might be carried out.

That said, a roadmap is not a necessary prerequisite for us supporting a project. In fact, part of the value we can add is to help you define a roadmap beyond the initial MVP, to ensure the best use of your time, talent and resources as you take your idea to fruition.              

Q9. What’s the open source / licensing model?          

This is the big one. The single precondition to a successful eLife Innovation Initiative collaboration is that the founders commit to keeping the core of their technology open in perpetuity, and license it accordingly.

While this does not preclude the monetization of the technology, it does go some way to preventing the loss of the core ideas and innovations behind the technology should its original implementation not be successful.

And since many of the projects we deal with concern scholarly content, keeping the core technologies behind the publishing, enhancement and discovery of such content open can also help avoid vendor lock-in and ensure the persistence of the content itself for archiving and reuse, no matter what happens to the original technology implementation. The last thing we want is for more scholarly content to be locked behind a proprietary technology.            

Q10. What do you expect from us?          

Just as we want to fully understand the background to a project idea and its founders, we also want to understand your expectations of us.

We work in a collaborative fashion, where any funding, resources, mentorship or exposure we provide in order to support a project are defined in close collaboration with you, and tailored as much as possible to your goals and expectations.
It is therefore important for us to ensure that, as we set expectations of you in the event of a successful collaboration agreement, we also set expectations of ourselves in the context of your vision.

Q11. Are you open to our input?          

In the spirit of setting expectations and  providing the best value in an eventual collaboration, we also need to understand your position with regards to input. Some teams come to us looking for direction and mentorship as well as funding for their project ideas, while some would prefer our contributions be limited to funding and exposure.

Provided we’re excited about your idea, and have satisfied ourselves of your ability to execute on it, we’re usually happy to comply with either scenario – but we find it’s best to establish those rules of engagement from the beginning, to set the right expectations on both sides and avoid any eventual misunderstandings.              

Q12. What’s the timeline for the project?            

When we talk about project timelines, we’re not talking about duration so much as sequence of events. Time, especially as it concerns fledgling technology projects, can sometimes be a “how long is a piece of string?” question, given that the timeframe of a project’s execution and roadmap is usually a result of a combination of variables including team composition, resourcing levels, funding availability and technology dependencies.

That said, it’s important to establish a mutual set of initial expectations, from our perspective as well as yours, in order to start defining the best way to approach both initial and ongoing support.

When we talk about project timelines, particularly with early-stage projects, we’re not talking about duration so much sequence of events. Is this a one-time funding? Is the funding linked to milestones? What happens after launch? What’s the awareness and exposure plan? Of course, we fully expect a project timeline to change based on the need to react to changing conditions, but we like to have a solid starting point to which we can aspire.

Establishing a shared sense of how the project might evolve over time upfront is an important part of establishing a solid foundation for its execution, ensuring the right expectations are set on both sides of the project.            

In summary          

eLife’s way of evaluating ideas is by no means unique. While this summary aims to elucidate some of the thinking behind the questions we pose when evaluating a project proposal, it should not be seen as a checklist for a successful funding proposal. Rather, we hope these questions and the thinking behind them can be used by project leads and teams as loose guidelines when thinking about how they (you?) plan to communicate their (your?) idea. If nothing else, a thoughtful approach to these questions should result in a more fleshed-out, concrete project proposal while avoiding some of the pitfalls of early-stage pitches.

This blogpost was prepared during the Open Source Alliance for Open Science’s Handbook Hackathon (July 2018) and was originally posted on eLife Innovation's blog. We welcome comments, questions and feedback. Please annotate publicly on the article or contact us at innovation [at] elifesciences [dot] org.For the latest in innovation, eLife Labs and new open-source tools, sign up for our technology and innovation newsletter. You can also follow @eLifeInnovation on Twitter.