For a long time that I am against the creation of Journal and Conference indexes, trying to stamp the contributions published or presented on those venues as good or bad. While I agree that there are some conferences that accept mostly every document that they receive, that does not mean that, a distracted researcher, could try and publish there some great work. This is the main problem of statistics altogether, as they consider the whole for the part.
Some other examples of the problems on this kind of journal ranking can be given. Depending on the area you do your research on, there are very different number of venues to publish your work. Thus, a couple researchers will have a simpler task while others will have an impossible task.
While there are topics that can be published in a wide range of journals, there are some other subjects that are too specialized, that there isn’t so much reference places where to publish it. Of course the researcher can publish the work in a less specific journal. There are some wide subject journals that have a good ranking. But then, would it be better to publish your work on some place where the standard researcher will not search for work on a specific topic, or publish in a less reputable conference where all the community publishes?
If this seems all too abstract, I can give an example. There aren’t much top ranking journals to publish on Natural Language Processing (NLP). Even for conferences, there are just a few with a good ranking. But then, if you look to any paper on NLP, I can be mostly sure that you can find, at least, one reference to a paper published in the Languages, Resources and Evaluation Conference (LREC). This one, is not top ranked. Also, they have a quite small rejection percentage. But all main researchers of the area publish there. It is a huge conference, with more than 700 papers accepted each year.
But, even if you accept that journals and conferences should be rated, and that a researcher work should be evaluated by such rankings, then you must look to what these rating organizations do. Let me give you a small example: I found that a journal I co-edit is present on SciMago (here). The problem is that, although the name and ISSN is correct, the subject classification is a little at the side and, worse, all the information on the publisher is wrong. SciMago claim that this information comes from Scopus, and that they just process it. But after contacting a couple of times Scopus, they aren’t able to fix the data. They redirect me to SciMago. When contacting SciMago, they route the conversation to Scopus again. In order to show Scopus that this is their problem, I tried to ask SciMago to reply to my mail including Scopus in the recipients. They just replied me, with the same copy&paste text from before.
If these services aren’t able to fix their data, even when asked for, can we trust on all other data they claim to have?