Ticket of the month - March 2022 - Getting started with REST API queries

[ifarley]

Hi @psmku,

For the query:

https://0-api-crossref-org.lib.rivier.edu/works?query.bibliographic=LGBT+LGBTQ+LGBTQI+LGBTQIA&filter=type:book-chapter,type:journal-article&select=DOI,title,subtitle,container-title,type,ISBN,ISSN,publisher,publisher-location,published-print,published-online,is-referenced-by-count,references-count,abstract,subject&rows=1000&offset=0

there are over 10,000 results. Offsets for /works are limited to 10,000, so using offsets here is not recommended. Instead, you should be using cursor. See more details on this distinction below.

Offset

The number of returned items is controlled by the rows parameter, but you can select the offset into the result list by using the offset parameter. So, for example, to select the second set of 5 results (i.e. results 6 through 10), you would do the following:

https://0-api-crossref-org.lib.rivier.edu/works?query=allen+renear&rows=5&offset=5

Offsets for /works are limited to 10K. Use cursor (see below) for larger /works results sets.

Deep paging with cursors

Using large offset values can result in extremely long response times. Offsets in the 100,000s and beyond will likely cause a timeout before the API is able to respond. An alternative to paging through very large result sets (like a corpus used for text and data mining) is to use the API’s exposure of Solr’s deep paging cursors. Any combination of query, filters and facets may be used with deep paging cursors. While rows may be specified along with cursor, offset and sample cannot be used. To use deep paging make a query as normal, but include the cursor parameter with a value of *. In this example we will page through all journal-article works from member 311:

https://0-api-crossref-org.lib.rivier.edu/members/311/works?filter=type:journal-article&cursor=*

A next-cursor field will be provided in the JSON response. To get the next page of results, pass the value of next-cursor as the cursor parameter. For example:

https://api.crossref.org/members/311/works?filter=type:journal-article&cursor=AoE/CGh0dHA6Ly9keC5kb2kub3JnLzEwLjEwMDIvdGRtX2xpY2Vuc2VfMQ==

Note that the actual cursor value will be different from this illustration.

Clients should check the number of returned items. If the number of returned items is fewer than the number of expected rows then the end of the result set has been reached. Using next-cursor beyond this point will result in responses with an empty items list.

The cursor parameter is available on all /works resources.

My best,
Isaac

[AlRen]

Hi @ifarley
I wonder if it is possible to get access to all the meta data of articles published in a specific journal. I would like to do a bibliometrics analysis of a journal (journal of applied psychology - ISSN: 1939-1854), but if I manage to get access to journal info and sometimes to few articles published in it, I don’t get the whole set of publications. Could you help me or at least guide me on how to build a relevant query regarding my research goal. Note that I am relatively neophyte into coding and API manipulating… and I am sorry for that if my request is irrelevant to you
Thanks.
Alex

[ifarley]

Hi @AlRen ,

If the question is relevant to you, then it is relevant to me and others in our community.

Here is the query you’d use to retrieve all the full metadata records that have been registered that include the ISSN 1939-1854:

https://0-api-crossref-org.lib.rivier.edu/journals/1939-1854/works?mailto=support@crossref.org

There are 10,175 DOIs registered with us that include that ISSN in the metadata record.

Additionally, you may review the depositor report for the Journal of Applied Psychology here:
https://0-data-crossref-org.lib.rivier.edu/depositorreport?pubid=J3371

This report includes all of the DOIs that have been registered for this journal, the DOI prefix that owns each DOI, the date of the last metadata update, and citation counts for each individual DOI.

My best,
Isaac

[AlRen]

Hi @ifarley,
Thank you for your quick answer and help.
I ve tried the request you suggested me and it works (I added &rows=1000&offset=0 because otherwise I retrieved only 56 docs). I manage to download the metadata of articles published in this journal, except the cited references, which is the category we are working on with our bibliometric methods.
Do you know why it is possible to dl the cited references when I focus on one specific articles metadata and not when I want to download a full set of metadata?
Thanks a lot for your help.
Alex

[ifarley]

Hi @AlRen ,

Thanks for following up. I think this query might give you everything you’re looking for. I’ve tried to reduce some of the noise and only select for elements of interest. So, at the time of this writing, there are only 667 DOIs registered for this journal that include reference metadata. The remaining DOIs have not had reference metadata registered for them.

https://0-api-crossref-org.lib.rivier.edu/journals/1939-1854/works?mailto=support@crossref.org&filter=has-references:true&select=DOI,title,type,reference,references-count,is-referenced-by-count&rows=700

My best,
Isaac

[AlRen]

Thanks again @ifarley. I downloaded the 667 metadata entries.
May I ask you two more questions and a last request.

1/ could you decrypt the request you wrote, I mean to explain the logic of it? Then I can reproduce it later fully aware of what I do.

Q1: Is it normal that the cited references don’t have the same form. indeed, some are cited this way:

{“doi-asserted-by”:“crossref”,“unstructured”:“Austin. Another view of dynamic criteria: A critical reanalysis of Barrett, Caldwell, and Alexander. 42 583 1989 », « key”:“r2_10.1037/0021-9010.86.3.446”,“DOI”:“10.1111/j.1744-6570.1989.tb00670.x”}

while other are cited this way:

{« doi-asserted-by":“publisher”,“key”:“r5_10.1037/0021-9010.86.3.446”,“DOI”:“10.2307/256190”},

I am used to use Scopus data and we collect raw data, exactly as they are cited by authors in their publications. Does crossref clean the cited references?

Q2: Do you have any idea why only 667 have references? Is it because the journal is poorly referenced and it doesn’t provide full data to Crossref? Or is it because most of its publication doesn’t cite any references? In both case the differential between 10K documents and 667 is impressive…

Thanks a lot.
Sincerely
Alex

[ifarley]

Hey @AlRen ,

See my answers below.

@AlRen

1/ could you decrypt the request you wrote, I mean to explain the logic of it? Then I can reproduce it later fully aware of what I do.

Sure thing.

https://0-api-crossref-org.lib.rivier.edu/journals/1939-1854/works?mailto=support@crossref.org&filter=has-references:true&select=DOI,title,type,reference,references-count,is-referenced-by-count&rows=700

I’ll go in order of the query, so you can follow the logic:

1. I’m querying the journals route for any works with ISSN 1939-1854.

2. By including my email address as a mailto parameter, my query is being performed by our Polite pool of the API.

3. I am then filtering the responses so that I only get results for DOIs with ISSN 1939-1854 in the metadata that have references registered with us.

4. Next, using the select parameter, I am telling the API to reduce some of the noise of the response, so I don’t get the full metadata record. Instead, I only want: DOI, title of the article registered for the DOI, content type of the DOI (these are all journal articles, so moot), the references registered for this DOI, the count of references registered for this DOI, and then the number of other Crossref DOIs that have cited this DOI (using the is-referenced-by-count parameter).

5. And, finally, since I know there are 667 results and I want to see all of the results in the response, I have told the API to give me the first 700 rows (or, resulting records) back.

Q1: Is it normal that the cited references don’t have the same form. indeed, some are cited this way:

{“doi-asserted-by”:“crossref”,“unstructured”:“Austin. Another view of dynamic criteria: A critical reanalysis of Barrett, Caldwell, and Alexander. 42 583 1989 », « key”:“r2_10.1037/0021-9010.86.3.446”,“DOI”:“10.1111/j.1744-6570.1989.tb00670.x”}

while other are cited this way:

{« doi-asserted-by":“publisher”,“key”:“r5_10.1037/0021-9010.86.3.446”,“DOI”:“10.2307/256190”},

This is somewhat normal. I suspect that whoever was registering this content for American Psychological Association (APA), the Crossref member who stewards the Journal of Applied Psychology (ISSN 1939-1854), had the citation for the DOI matching 10.1111/j.1744-6570.1989.tb00670.x, but did not know the DOI, so they submitted the citation and we matched the DOI, so that is why the reference metadata is presented in this way.

For the reference metadata where only the DOI is present, American Psychological Association (APA) only gave us the DOI to establish the cited-by match. They didn’t need to provide us with the citation because they had the DOI. That’s the most definitive way to establish a cited-by match between the citing and cited DOIs.

Does crossref clean the cited references?

No.

Q2: Do you have any idea why only 667 have references? Is it because the journal is poorly referenced and it doesn’t provide full data to Crossref? Or is it because most of its publication doesn’t cite any references? In both case the differential between 10K documents and 667 is impressive…

We have 667 DOIs with reference metadata for this ISSN because this is what American Psychological Association (APA) has registered with us. Unfortunately, the questions about the content and why only 667 DOIs have reference metadata registered are best answered by APA. I can tell you that we do not require our members to register reference metadata, so it is likely that there is reference metadata for DOIs of this journal that just have not (yet) been registered with us. Adding reference metadata to any existing Crossref DOI’s metadata record is free for our members, but they’d need to register the references.

Warm regards,
Isaac

[AlRen]

Hello @ifarley
Thank you for all your answers and advice. They were very informative for me and helped me a lot. I have a broader question, not sure if this is the right place to ask it, but anyway. I wanted to explore CrossRef’s coverage of journals referenced in a scientific ranking. I found the right command to explore a single journal, but is it possible to get the list of journals present in CrossRef, or at least the ISSN list? And if so, what information can be added: for example, would it be possible to have the years covered and the number of articles included in each journal?
Thank you for your help, which is always valuable.
regards
Alex

[ifarley]

Hi Alex,

We don’t have queries that are going to give you exactly what you are requesting, but here are a few that get close/are a starting point:

This will give you all journal articles registered with us (I have selected for only the article DOI, article title, journal title, and ISSN):
https://0-api-crossref-org.lib.rivier.edu/works?filter=type:journal-article&select=DOI,title,container-title,ISSN&rows=1000&mailto=support@crossref.org

Similar to the first query, this gives you journal articles sorted by date created (or, registered with Crossref). The newest DOIs are atop the results:
https://0-api-crossref-org.lib.rivier.edu/works?sort=created&filter=type:journal-article&select=DOI,ISSN,container-title,created&rows=1000&mailto=support@crossref.org

If you know the ISSN you’re eager to see works for you, you can include it in your query, like this one below. These results are sorted in order of most recently created (registered with Crossref):
https://0-api-crossref-org.lib.rivier.edu/works?sort=created&filter=issn:1939-1854&select=DOI,title,container-title,created&rows=1000&mailto=support@crossref.org

My best,
Isaac

[AlRen]

Hi Isaac,
I am back here with another question ^^
Do you know if it is possible to have the list of metadata available for a specific journal / or specific editor?
I have 2 specific concerns:

• does the journal/editor provide: (i) the abstracts; (ii) the references cited

I will be interested for this journal Accounting, Organizations and Society (Print ISSN: 1873-6289 and e-ISSN: 0361-3682); and this editor; Springer with the prefix 10.1016

Thanks a lot.
Regards,
Alexandre

[ifarley]

Hi @AlRen ,

Good questions. You can retrieve this information in our REST API:

There are no works registered with us that include ISSN 1873-6289 in the metadata record, as you can see with this query: https://0-api-crossref-org.lib.rivier.edu/journals/18736289/works?mailto=support@crossref.org

As for works registered with us for ISSN 0361-3682, some have references registered:
https://0-api-crossref-org.lib.rivier.edu/journals/03613682/works?filter=has-references:true,&mailto=support@crossref.org

But, none have abstracts registered with us:
https://0-api-crossref-org.lib.rivier.edu/journals/03613682/works?filter=has-abstract:true,&mailto=support@crossref.org

You can confirm that here (all works are returned when including the parameter: filter=has-abstract:false)
https://0-api-crossref-org.lib.rivier.edu/journals/03613682/works?filter=has-abstract:false,&mailto=support@crossref.org

Which works have an abstract registered with us on prefix 10.1016:
https://api.crossref.org/prefixes/10.1016/works?filter=has-abstract:true&mailto=support@crossref.org

Which works have references registered with us on prefix 10.1016:
https://0-api-crossref-org.lib.rivier.edu/prefixes/10.1016/works?filter=has-references:true&mailto=support@crossref.org

My best,
Isaac

[AlRen]

Thank you so much for your precise and so helpful answers.
I think I have everything I need. Thank you so much again

[ifarley]

You’re welcome! We’re always happy to help

[edgolovin]

Hello, Isaac,

in your first message in this thread you wrote an example of a query, which processes the query tokens with OR:

@ifarley

I always like starting with a straightforward query like this:
works?query.affiliation=Science+State+University&select=DOI,title,author&rows=500&mailto=support@crossref.org)

In this query, I am searching our whole corpus for any affiliation that includes the word Science or State or University;

My question is, how to make a query with AND operand between several tokens?

[ifarley]

Hi @edgolovin ,

Thanks for following up.

You’re right that in the query: https://0-api-crossref-org.lib.rivier.edu/works?query.affiliation=Science+State+University&select=DOI,title,author&rows=500&mailto=support@crossref.org that I get back all results that include the word Science OR State OR University (that’s why I get back more than 12 million results), but the results are returned in order of their relevance for Science AND State AND University. That relevance is scored using a relevance score, you can see it if you include it in the select parameter, like this:

https://0-api-crossref-org.lib.rivier.edu/works?query.affiliation=Science+State+University&select=DOI,title,author,score&rows=500&mailto=support@crossref.org

My best,
Isaac

[edgolovin]

Thank you, @ifarley.

Your answer is very clear and helpful. I will take in use this relevance score then.

Could you point out if there is any documentation on how this score is calculated, or maybe the particular name of the relevance model?

[ifarley]

Hi @edgolovin ,

We use the default scoring in Elasticsearch (ES): Practical BM25 - Part 2: The BM25 Algorithm and its Variables | Elastic Blog. The query is scored against a field which is a concatenation of several metadata fields.

There is also one additional thing to keep in mind about these scores: in search engines, such as ES, scoring is not supposed to be meaningful across different queries, i.e. the score is not some sort of objective global measure of similarity. The number is not scaled to any known range, and it will depend a lot on the query itself. Scores are only supposed to allow us to compare the similarity of different indexed documents with the same query , and so it only enables us to sort the results for a given query.

My best,
Isaac

[AlRen]

hi Isaac,
I am wondering ig it is possible to write a query on the keywords used by articles’ authors ? I can’t find the command.
Thank you by advance for your help
sincerely

[ifarley]

Hi @AlRen ,

Unfortunately, keywords asserted by authors are not a part of the metadata record registered with Crossref, so there’s no way to search for an author’s keywords using our API.

-Isaac

[ifarley]

Hello all,

For those of you looking to get started with using our API, I highly recommend the API 101 for publishers, researchers, and librarians with Postman and Crossref - ConTech.Live video tutorial hosted by my colleague Rachael Lammey, our Director of Product, and Claire Froelich, Student Community Manager at Postman. So well done!

-Isaac