added a new working table to import the suggestions

Reviewed-on: #20

LICENSE

@@ -0,0 +1,661 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU Affero General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU AGPL, see
+<http://www.gnu.org/licenses/>.

apps/bioschemas-api/pom.xml

@@ -3,7 +3,7 @@
     <parent>
         <groupId>eu.dnetlib.dhp</groupId>
         <artifactId>apps</artifactId>
-        <version>3.3.3</version>
+        <version>3.5.5-SNAPSHOT</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

apps/dhp-broker-application/pom.xml

@@ -4,7 +4,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>apps</artifactId>
-		<version>3.3.3</version>
+		<version>3.5.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 

apps/dhp-broker-application/src/main/resources/application.properties

@@ -51,10 +51,9 @@ lbs.mail.smtpUser = smtp-dnet
 lbs.mail.smtpPassword = hhr*7932
 lbs.mail.message.template = classpath:/templates/dhp_openaire_mail.st
 lbs.mail.message.template.dashboard.baseUrl  = https://provide.openaire.eu/content/notifications/
-lbs.mail.message.template.dashboard.helpdesk = https://www.openaire.eu/support/helpdesk
+lbs.mail.message.template.dashboard.helpdesk = https://www.openaire.eu/helpdesk
 lbs.mail.message.template.publicApi = https://api.openaire.eu/broker
 
-
 lbs.queues.maxReturnedValues = 1000
 
 lbs.task.deleteOldEvents.cron           = 0 0 4 1/1 * ?

apps/dhp-broker-public-application/pom.xml

@@ -4,7 +4,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>apps</artifactId>
-		<version>3.3.3</version>
+		<version>3.5.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 

apps/dhp-broker-public-application/src/main/java/eu/dnetlib/broker/oa/controllers/EventFeedback.java

@@ -2,6 +2,8 @@ package eu.dnetlib.broker.oa.controllers;
 
 import java.io.Serializable;
 
+import eu.dnetlib.broker.common.feedbacks.FeedbackStatus;
+
 public class EventFeedback implements Serializable {
 
 	/**
@@ -10,27 +12,22 @@ public class EventFeedback implements Serializable {
 	private static final long serialVersionUID = -6967719685282712195L;
 
 	private String eventId;
-	private String status;
 
-	// TOOD status should be an enum having the following values:
-	// * DISCARDED: the event was not processable by the system. OpenAIRE should not interpret such status in a negative or positive sense
-	// with regard to the accuracy of the notification
-	// * REJECTED: a human takes the decision to reject the suggestion as it was wrong
-	// * ACCEPTED: a human takes the decision to apply the suggested enrichment to the local record
+	private FeedbackStatus status;
 
-	protected String getEventId() {
+	public String getEventId() {
 		return eventId;
 	}
 
-	protected void setEventId(final String eventId) {
+	public void setEventId(final String eventId) {
 		this.eventId = eventId;
 	}
 
-	protected String getStatus() {
+	public FeedbackStatus getStatus() {
 		return status;
 	}
 
-	protected void setStatus(final String status) {
+	public void setStatus(final FeedbackStatus status) {
 		this.status = status;
 	}
 

apps/dhp-broker-public-application/src/main/java/eu/dnetlib/broker/oa/controllers/OpenairePublicController.java

@@ -57,6 +57,8 @@ import eu.dnetlib.broker.api.ShortEventMessage;
 import eu.dnetlib.broker.common.elasticsearch.EventRepository;
 import eu.dnetlib.broker.common.elasticsearch.Notification;
 import eu.dnetlib.broker.common.elasticsearch.NotificationRepository;
+import eu.dnetlib.broker.common.feedbacks.DbEventFeedback;
+import eu.dnetlib.broker.common.feedbacks.DbEventFeedbackRepository;
 import eu.dnetlib.broker.common.properties.ElasticSearchProperties;
 import eu.dnetlib.broker.common.stats.OpenaireDsStatRepository;
 import eu.dnetlib.broker.common.subscriptions.Subscription;
@@ -87,6 +89,9 @@ public class OpenairePublicController extends AbstractDnetController {
 	@Autowired
 	private OpenaireDsStatRepository openaireDsStatRepository;
 
+	@Autowired
+	private DbEventFeedbackRepository feedbackRepository;
+
 	@Autowired
 	private ElasticSearchProperties props;
 
@@ -103,31 +108,27 @@ public class OpenairePublicController extends AbstractDnetController {
 
 		final Optional<Subscription> optSub = subscriptionRepo.findById(subscrId);
 
-		if (optSub.isPresent()) {
+		if (!optSub.isPresent()) {
+			log.warn("Invalid subscription: " + subscrId);
+			return new ScrollPage<>(null, true, new ArrayList<>());
+		}
+		final ElasticsearchRestTemplate esTemplate = (ElasticsearchRestTemplate) esOperations;
 
-			final ElasticsearchRestTemplate esTemplate = (ElasticsearchRestTemplate) esOperations;
-
-			final NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
+		final NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
 				.withQuery(QueryBuilders.termQuery("subscriptionId.keyword", subscrId))
 				.withSearchType(SearchType.DEFAULT)
 				.withFields("topic", "payload", "eventId")
 				.withPageable(PageRequest.of(0, 100))
 				.build();
 
-			final SearchScrollHits<Notification> scroll =
+		final SearchScrollHits<Notification> scroll =
 				esTemplate.searchScrollStart(SCROLL_TIMEOUT_IN_MILLIS, searchQuery, Notification.class, IndexCoordinates.of(props.getNotificationsIndexName()));
-			if (scroll.hasSearchHits()) {
-				final List<ShortEventMessage> values = calculateNotificationMessages(scroll);
-				return new ScrollPage<>(scroll.getScrollId(), values.isEmpty() || scroll.getScrollId() == null, values);
-			} else {
-				esTemplate.searchScrollClear(Arrays.asList(scroll.getScrollId()));
-				return new ScrollPage<>(null, true, new ArrayList<>());
-			}
-
-		} else {
-			log.warn("Invalid subscription: " + subscrId);
-			return new ScrollPage<>(null, true, new ArrayList<>());
+		if (scroll.hasSearchHits()) {
+			final List<ShortEventMessage> values = calculateNotificationMessages(scroll);
+			return new ScrollPage<>(scroll.getScrollId(), values.isEmpty() || scroll.getScrollId() == null, values);
 		}
+		esTemplate.searchScrollClear(Arrays.asList(scroll.getScrollId()));
+		return new ScrollPage<>(null, true, new ArrayList<>());
 	}
 
 	@Operation(summary = "Returns notifications using scrolls (other pages)")
@@ -137,14 +138,13 @@ public class OpenairePublicController extends AbstractDnetController {
 		final ElasticsearchRestTemplate esTemplate = (ElasticsearchRestTemplate) esOperations;
 
 		final SearchScrollHits<Notification> scroll =
-			esTemplate.searchScrollContinue(scrollId, SCROLL_TIMEOUT_IN_MILLIS, Notification.class, IndexCoordinates.of(props.getNotificationsIndexName()));
+				esTemplate.searchScrollContinue(scrollId, SCROLL_TIMEOUT_IN_MILLIS, Notification.class, IndexCoordinates.of(props.getNotificationsIndexName()));
 		if (scroll.hasSearchHits()) {
 			final List<ShortEventMessage> values = calculateNotificationMessages(scroll);
 			return new ScrollPage<>(scroll.getScrollId(), values.isEmpty() || scroll.getScrollId() == null, values);
-		} else {
-			esTemplate.searchScrollClear(Arrays.asList(scroll.getScrollId()));
-			return new ScrollPage<>(null, true, new ArrayList<>());
 		}
+		esTemplate.searchScrollClear(Arrays.asList(scroll.getScrollId()));
+		return new ScrollPage<>(null, true, new ArrayList<>());
 	}
 
 	@Operation(summary = "Returns notifications as file")
@@ -197,8 +197,8 @@ public class OpenairePublicController extends AbstractDnetController {
 		final Path pathDir = new Path(opendoarEventsPath + "/" + DigestUtils.md5Hex(id));
 
 		try (final FileSystem fs = FileSystem.get(conf);
-			final ServletOutputStream out = res.getOutputStream();
-			final GZIPOutputStream gzOut = new GZIPOutputStream(out)) {
+				final ServletOutputStream out = res.getOutputStream();
+				final GZIPOutputStream gzOut = new GZIPOutputStream(out)) {
 			boolean first = true;
 
 			IOUtils.write("[\n", gzOut, StandardCharsets.UTF_8);
@@ -209,16 +209,16 @@ public class OpenairePublicController extends AbstractDnetController {
 						final Path path = fileStatus.getPath();
 						if (path.getName().endsWith(".json")) {
 							try (final FSDataInputStream fis = fs.open(path);
-								final InputStreamReader isr = new InputStreamReader(fis);
-								final BufferedReader br = new BufferedReader(isr)) {
+									final InputStreamReader isr = new InputStreamReader(fis);
+									final BufferedReader br = new BufferedReader(isr)) {
 
 								first = processLine(gzOut, first, br);
 							}
 						} else if (path.getName().endsWith(".json.gz")) {
 							try (final FSDataInputStream fis = fs.open(path);
-								final GZIPInputStream gzIn = new GZIPInputStream(fis);
-								final InputStreamReader isr = new InputStreamReader(gzIn);
-								final BufferedReader br = new BufferedReader(isr)) {
+									final GZIPInputStream gzIn = new GZIPInputStream(fis);
+									final InputStreamReader isr = new InputStreamReader(gzIn);
+									final BufferedReader br = new BufferedReader(isr)) {
 
 								first = processLine(gzOut, first, br);
 							}
@@ -271,24 +271,30 @@ public class OpenairePublicController extends AbstractDnetController {
 
 	@Operation(summary = "Store the feedback of an event (MOCK)")
 	@RequestMapping(value = "/feedback/events", method = {
-		RequestMethod.POST, RequestMethod.PATCH
+			RequestMethod.POST, RequestMethod.PATCH
 	})
-	private Map<String, String> feedbackEvent(@RequestBody final EventFeedback feedback) {
-		// TOOD
-		final Map<String, String> res = new HashMap<>();
+	private Map<String, Object> feedbackEvent(@RequestBody final EventFeedback feedback) {
+
+		final DbEventFeedback dbEntry = new DbEventFeedback();
+		dbEntry.setEventId(feedback.getEventId());
+		dbEntry.setStatus(feedback.getStatus());
+
+		feedbackRepository.save(dbEntry);
+
+		final Map<String, Object> res = new HashMap<>();
 		res.put("status", "done");
+		res.put("feedback", dbEntry);
 		return res;
 	}
 
 	private List<ShortEventMessage> calculateNotificationMessages(final SearchScrollHits<Notification> scroll) {
 		if (scroll.getSearchHits().size() > 0) {
 			return scroll.stream()
-				.map(SearchHit::getContent)
-				.map(this::messageFromNotification)
-				.collect(Collectors.toList());
-		} else {
-			return new ArrayList<>();
+					.map(SearchHit::getContent)
+					.map(this::messageFromNotification)
+					.collect(Collectors.toList());
 		}
+		return new ArrayList<>();
 	}
 
 	private ShortEventMessage messageFromNotification(final Notification n) {

apps/dhp-broker-public-application/ssh_tunnel_postgres.sh

@@ -1,3 +0,0 @@
-#!/bin/bash
-
-ssh -vNL 5432:10.19.65.40:5432 michele.artini@iis-cdh5-test-gw.ocean.icm.edu.pl

apps/dhp-mdstore-manager/pom.xml

@@ -4,7 +4,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>apps</artifactId>
-		<version>3.3.3</version>
+		<version>3.5.5-SNAPSHOT</version>
 		<relativePath>../pom.xml</relativePath>
 	</parent>
 

apps/dnet-exporter-api/pom.xml

@@ -3,7 +3,7 @@
 	<parent>
 		<groupId>eu.dnetlib.dhp</groupId>
 		<artifactId>apps</artifactId>
-		<version>3.3.3</version>
+		<version>3.5.5-SNAPSHOT</version>
 		<relativePath>../</relativePath>
 	</parent>
 
@@ -26,6 +26,11 @@
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-cache</artifactId>
 		</dependency>
+		<dependency>
+			<groupId>eu.dnetlib.dhp</groupId>
+			<artifactId>dnet-exporter-model</artifactId>
+			<version>${project.version}</version>
+		</dependency>
 		<dependency>
 			<groupId>eu.dnetlib</groupId>
 			<artifactId>cnr-rmi-api</artifactId>
@@ -108,6 +113,10 @@
 			<groupId>com.fasterxml.jackson.datatype</groupId>
 			<artifactId>jackson-datatype-joda</artifactId>
 		</dependency>
+		<dependency>
+			<groupId>com.fasterxml.jackson.datatype</groupId>
+			<artifactId>jackson-datatype-jsr310</artifactId>
+		</dependency>
 		<dependency>
 			<groupId>joda-time</groupId>
 			<artifactId>joda-time</artifactId>
@@ -125,13 +134,24 @@
 			<artifactId>dnet-datasource-manager-common</artifactId>
 			<version>[2.0.1,3.0.0)</version>
 		</dependency>
-
+		
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-test</artifactId>
 			<scope>test</scope>
 		</dependency>
-	</dependencies>
+        <dependency>
+            <groupId>org.springframework</groupId>
+            <artifactId>spring-web</artifactId>
+            <version>5.3.8</version>
+        </dependency>
+        
+		<dependency>
+			<groupId>com.vladmihalcea</groupId>
+			<artifactId>hibernate-types-52</artifactId>
+		</dependency>
+        
+    </dependencies>
 	
 	<build>
 		<plugins>

apps/dnet-exporter-api/src/main/java/eu/dnetlib/CacheCustomizer.java

@@ -1,29 +1,18 @@
 package eu.dnetlib;
 
+import static java.util.Arrays.asList;
+
 import org.springframework.boot.autoconfigure.cache.CacheManagerCustomizer;
 import org.springframework.cache.concurrent.ConcurrentMapCacheManager;
 import org.springframework.stereotype.Component;
 
-import static java.util.Arrays.asList;
-
 @Component
 public class CacheCustomizer implements CacheManagerCustomizer<ConcurrentMapCacheManager> {
 
-    @Override
-    public void customize(final ConcurrentMapCacheManager cacheManager) {
-        cacheManager.setCacheNames(
-                asList(
-                        "fundingpath-ids",
-                        "indexdsinfo-cache",
-                        "objectstoreid-cache",
-                        "context-cache",
-                        "context-cache-funder",
-                        "context-cache-community",
-                        "dsm-aggregationhistory-cache",
-                        "dsm-firstharvestdate-cache",
-                        "vocabularies-cache",
-                        "community-cache",
-                        "info"));
-    }
+	@Override
+	public void customize(final ConcurrentMapCacheManager cacheManager) {
+		cacheManager
+			.setCacheNames(asList("fundingpath-ids", "indexdsinfo-cache", "objectstoreid-cache", "context-cache", "context-cache-funder", "context-cache-community", "dsm-aggregationhistory-cache-v1", "dsm-aggregationhistory-cache-v2", "dsm-firstharvestdate-cache", "vocabularies-cache", "community-cache", "info"));
+	}
 
-}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/common/AbstractExporterController.java

@@ -16,10 +16,10 @@ import org.springframework.web.bind.annotation.ResponseStatus;
 
 import com.fasterxml.jackson.annotation.JsonAutoDetect;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.enabling.datasources.common.DsmForbiddenException;
-import eu.dnetlib.enabling.datasources.common.DsmNotFoundException;
-import eu.dnetlib.openaire.dsm.domain.Response;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.exceptions.ResourceNotFoundException;
+import eu.dnetlib.openaire.exporter.model.dsm.Response;
 
 /**
  * Created by claudio on 18/07/2017.
@@ -30,33 +30,26 @@ public abstract class AbstractExporterController {
 
 	@ResponseBody
 	@ExceptionHandler({
-		DsmException.class
+		DsmApiException.class, CommunityException.class, Exception.class
 	})
 	@ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
-	public ErrorMessage handleDSMException(final Exception e) {
-		return _handleError(e);
-	}
-
-	@ResponseBody
-	@ExceptionHandler(DsmForbiddenException.class)
-	@ResponseStatus(value = HttpStatus.FORBIDDEN)
-	public ErrorMessage handleForbiddenException(final Exception e) {
+	public ErrorMessage handle500(final Exception e) {
 		return _handleError(e);
 	}
 
 	@ResponseBody
 	@ExceptionHandler({
-		DsmNotFoundException.class
+		ResourceNotFoundException.class
 	})
 	@ResponseStatus(value = HttpStatus.NOT_FOUND)
-	public ErrorMessage handleNotFoundException(final Exception e) {
+	public ErrorMessage handle404(final Exception e) {
 		return _handleError(e);
 	}
 
 	@ResponseBody
 	@ExceptionHandler(MethodArgumentNotValidException.class)
 	@ResponseStatus(HttpStatus.BAD_REQUEST)
-	public List<ErrorMessage> processValidationError(final MethodArgumentNotValidException e) {
+	public List<ErrorMessage> handle400(final MethodArgumentNotValidException e) {
 		return e.getBindingResult()
 			.getFieldErrors()
 			.stream()

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/common/ExporterConstants.java

@@ -10,6 +10,7 @@ public class ExporterConstants {
 	public final static String C_PJ = "Community projects";
 	public final static String C_ZC = "Community Zenodo Communities";
 	public final static String C_O = "Community Organizations";
+	public final static String C_SUB = "Subcommunities";
 
 	public final static String DS = "Datasource";
 	public final static String API = "Interface";

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/common/ISClient.java

@@ -4,28 +4,35 @@ import java.io.IOException;
 import java.util.List;
 import java.util.Map;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.openaire.context.Context;
+import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpException;
 import eu.dnetlib.openaire.dsm.dao.utils.IndexDsInfo;
+import eu.dnetlib.openaire.exporter.model.context.Context;
 
 public interface ISClient {
 
-	IndexDsInfo calculateCurrentIndexDsInfo() throws DsmException;
+	IndexDsInfo calculateCurrentIndexDsInfo() throws Exception;
 
-	String getObjectStoreId(String dsId) throws DsmException;
+	String getObjectStoreId(String dsId) throws Exception;
 
+	@Deprecated
 	Map<String, Context> getFunderContextMap() throws IOException;
 
+	@Deprecated
 	Map<String, Context> getCommunityContextMap() throws IOException;
 
+	@Deprecated
 	Map<String, Context> getContextMap(final List<String> type) throws IOException;
 
-	void updateContextParam(String id, String name, String value);
+	@Deprecated
+	void updateContextParam(String id, String name, String value, boolean toEscape);
 
+	@Deprecated
 	void updateContextAttribute(String id, String name, String value);
 
+	@Deprecated
 	void addConcept(String id, String categoryId, String data);
 
+	@Deprecated
 	void removeConcept(String id, String categoryId, String conceptId);
 
 	void dropCache();
@@ -39,10 +46,15 @@ public interface ISClient {
 	 * @param value
 	 *            new value for the attribute
 	 */
+	@Deprecated
 	void updateConceptAttribute(String id, String name, String value);
 
+	@Deprecated
 	void updateConceptParam(String id, String name, String value);
 
+	@Deprecated
 	void updateConceptParamNoEscape(String id, String name, String value);
 
+	String getProfile(String profileId) throws ISLookUpException;
+
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/common/ISClientImpl.java

@@ -32,9 +32,9 @@ import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.enabling.datasources.common.DsmRuntimeException;
 import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpException;
 import eu.dnetlib.enabling.is.lookup.rmi.ISLookUpService;
-import eu.dnetlib.openaire.context.Context;
 import eu.dnetlib.openaire.context.ContextMappingUtils;
 import eu.dnetlib.openaire.dsm.dao.utils.IndexDsInfo;
+import eu.dnetlib.openaire.exporter.model.context.Context;
 
 /**
  * Created by claudio on 20/10/2016.
@@ -52,7 +52,7 @@ public class ISClientImpl implements ISClient {
 
 	@Override
 	@Cacheable("indexdsinfo-cache")
-	public IndexDsInfo calculateCurrentIndexDsInfo() throws DsmException {
+	public IndexDsInfo calculateCurrentIndexDsInfo() throws Exception {
 		log.warn("calculateCurrentIndexDsInfo(): not using cache");
 		final String[] arr;
 		try {
@@ -67,41 +67,49 @@ public class ISClientImpl implements ISClient {
 
 	@Override
 	@Cacheable("objectstoreid-cache")
-	public String getObjectStoreId(final String dsId) throws DsmException {
+	public String getObjectStoreId(final String dsId) throws Exception {
 		log.warn(String.format("getObjectStoreId(%s): not using cache", dsId));
 		try {
 			final String xqueryTemplate = _getQuery(config.getFindObjectStore());
 			return _isLookUp(String.format(xqueryTemplate, dsId));
 		} catch (IOException | ISLookUpException e) {
-			throw new DsmException("unble to find objectstore for ds " + dsId);
+			throw new DsmException("unable to find objectstore for ds " + dsId);
 		}
 	}
 
 	@Override
 	@Cacheable("context-cache-funder")
+	@Deprecated
 	public Map<String, Context> getFunderContextMap() throws IOException {
 		return _processContext(_getQuery(config.getFindFunderContexts()));
 	}
 
 	@Override
 	@Cacheable("context-cache-community")
+	@Deprecated
 	public Map<String, Context> getCommunityContextMap() throws IOException {
 		return _processContext(_getQuery(config.getFindCommunityContexts()));
 	}
 
 	@Override
 	@Cacheable("context-cache")
+	@Deprecated
 	public Map<String, Context> getContextMap(final List<String> type) throws IOException {
 		if (Objects.isNull(type) || type.isEmpty()) {
 			return _processContext(_getQuery(config.getFindContextProfiles()));
 		} else {
-			final String xqueryTemplate = _getQuery(config.getFindContextProfilesByType());
+			try {
+				final String xqueryTemplate = _getQuery(config.getFindContextProfilesByType());
+				final String xquery = String.format(xqueryTemplate, type.stream()
+					.map(t -> String.format("./RESOURCE_PROFILE/BODY/CONFIGURATION/context/@type = '%s'", t))
+					.collect(Collectors.joining(" or ")));
 
-			final String xquery = String.format(xqueryTemplate, type.stream()
-				.map(t -> String.format("./RESOURCE_PROFILE/BODY/CONFIGURATION/context/@type = '%s'", t))
-				.collect(Collectors.joining(" or ")));
+				return _processContext(xquery);
+			} catch (final Exception e) {
+				log.error(e.getMessage());
+				throw new RuntimeException(e);
+			}
 
-			return _processContext(xquery);
 		}
 	}
 
@@ -109,18 +117,41 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
-	public void updateContextParam(final String id, final String name, final String value) {
+	@Deprecated
+	public void updateContextParam(final String id, final String name, final String value, final boolean toEscape) {
+		if (getSize(id, name) > 0) {
+			try {
+				_quickSeachProfile(getXQuery(id, name, value, toEscape));
+			} catch (final ISLookUpException e) {
+				throw new DsmRuntimeException(String.format("unable to update context param [id: %s, name: %s, value: %s]", id, name, value), e);
+			}
+		} else {
+			try {
+				_quickSeachProfile(getInsertXQuery(id, name, value, toEscape));
+			} catch (final ISLookUpException e) {
+				throw new DsmRuntimeException(String.format("unable to insert context param [id: %s, name: %s, value: %s]", id, name, value), e);
+			}
+		}
+
+	}
+
+	private int getSize(final String id, final String name) {
+		int size = 0;
 		try {
-			_quickSeachProfile(getXQuery(id, name, value));
+			size = _quickSeachProfile(String
+				.format("for $x in  collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')  where $x//context[@id='%s']/param[@name='%s'] return $x", id, name))
+					.size();
 		} catch (final ISLookUpException e) {
-			throw new DsmRuntimeException(String.format("unable update context param [id: %s, name: %s, value: %s]", id, name, value), e);
+			throw new DsmRuntimeException("unable to execute search query", e);
 		}
+		return size;
 	}
 
 	@Override
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void updateContextAttribute(final String id, final String name, final String value) {
 		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
 		try {
@@ -135,6 +166,7 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void addConcept(final String id, final String categoryId, final String data) {
 		try {
 			_quickSeachProfile(String.format("update insert %s into collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
@@ -148,6 +180,7 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void removeConcept(final String id, final String categoryId, final String conceptId) {
 		try {
 			_quickSeachProfile(String.format("for $concept in collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
@@ -163,6 +196,7 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-community", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void updateConceptAttribute(final String id, final String name, final String value) {
 		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
 		try {
@@ -178,6 +212,7 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void updateConceptParam(final String id, final String name, final String value) {
 		try {
 			_quickSeachProfile(getConceptXQuery(id, name, value));
@@ -190,6 +225,7 @@ public class ISClientImpl implements ISClient {
 	@CacheEvict(value = {
 		"context-cache", "context-cache-funder"
 	}, allEntries = true)
+	@Deprecated
 	public void updateConceptParamNoEscape(final String id, final String name, final String value) {
 		try {
 			_quickSeachProfile(getConceptXQueryNoEscape(id, name, value));
@@ -200,17 +236,51 @@ public class ISClientImpl implements ISClient {
 
 	/// HELPERS
 
-	private String getXQuery(final String id, final String name, final String value) {
-		final Escaper esc = XmlEscapers.xmlContentEscaper();
+	@Deprecated
+	private String getInsertXQuery(final String id, final String paramName, final String paramValue, final boolean toEscape) {
+		String value;
+		if (toEscape) {
+			value = escape(XmlEscapers.xmlContentEscaper(), paramValue);
+		} else {
+			value = paramValue;
+		}
+		if (StringUtils.isNotBlank(value)) {
+			return String.format("update insert <param name='%s'>%s</param> into collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
+				"/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']", paramName, value, id);
+		} else {
+			return String.format("update insert <param name='%s'/> into collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
+				"/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']", paramName, id);
+		}
+	}
+
+	@Deprecated
+	private String getXQuery(final String id, final String name, final String paramValue, final boolean toEscape) {
+		String value = paramValue;
+		if (toEscape) {
+			value = escape(XmlEscapers.xmlContentEscaper(), paramValue);
+		}
+
 		if (StringUtils.isNotBlank(value)) {
 			return String.format("update replace collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
-				"/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']/param[./@name = '%s'] with <param name='%s'>%s</param>", id, name, name, escape(esc, value));
+				"/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']/param[./@name = '%s'] with <param name='%s'>%s</param>", id, name, name, value);
 		} else {
 			return String.format("update replace collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
 				"/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']/param[./@name = '%s'] with <param name='%s'/>", id, name, name);
 		}
 	}
 
+	// private String getXQueryNoEscape(final String id, final String name, final String value) {
+	// if (StringUtils.isNotBlank(value)) {
+	// return String.format("update replace collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
+	// "/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']/param[./@name = '%s'] with <param name='%s'>%s</param>", id, name, name,
+	// value);
+	// } else {
+	// return String.format("update replace collection('/db/DRIVER/ContextDSResources/ContextDSResourceType')" +
+	// "/RESOURCE_PROFILE/BODY/CONFIGURATION/context[./@id = '%s']/param[./@name = '%s'] with <param name='%s'/>", id, name, name);
+	// }
+	// }
+
+	@Deprecated
 	private String getConceptXQuery(final String id, final String name, final String value) {
 		final Escaper esc = XmlEscapers.xmlContentEscaper();
 		if (StringUtils.isNotBlank(value)) {
@@ -222,6 +292,7 @@ public class ISClientImpl implements ISClient {
 		}
 	}
 
+	@Deprecated
 	private String getConceptXQueryNoEscape(final String id, final String name, final String value) {
 
 		if (StringUtils.isNotBlank(value)) {
@@ -233,10 +304,12 @@ public class ISClientImpl implements ISClient {
 		}
 	}
 
+	@Deprecated
 	private Map<String, Context> _processContext(final String xquery) throws IOException {
 		return _processContext(new LinkedBlockingQueue<>(), xquery);
 	}
 
+	@Deprecated
 	private Map<String, Context> _processContext(final Queue<Throwable> errors, final String xquery) throws IOException {
 		try {
 			return getContextProfiles(errors, xquery).stream()
@@ -254,6 +327,7 @@ public class ISClientImpl implements ISClient {
 		}
 	}
 
+	@Deprecated
 	private List<String> getContextProfiles(final Queue<Throwable> errors, final String xquery) throws IOException {
 		log.warn("getContextProfiles(): not using cache");
 		try {
@@ -277,11 +351,17 @@ public class ISClientImpl implements ISClient {
 		final List<String> res = Lists.newArrayList();
 
 		log.debug(String.format("running xquery:\n%s", xquery));
-		final List<String> list = isLookUpService.quickSearchProfile(xquery);
-		if (list != null) {
-			res.addAll(list);
+		try {
+			final List<String> list = isLookUpService.quickSearchProfile(xquery);
+			if (list != null) {
+				res.addAll(list);
+			}
+			log.debug(String.format("query result size: %s", res.size()));
+		} catch (final Exception ex) {
+			log.error(ex.getMessage());
+			throw new ISLookUpException("");
 		}
-		log.debug(String.format("query result size: %s", res.size()));
+
 		return res;
 	}
 
@@ -294,4 +374,9 @@ public class ISClientImpl implements ISClient {
 		log.debug("dropped dsManager IS cache");
 	}
 
+	@Override
+	public String getProfile(final String profileId) throws ISLookUpException {
+		return isLookUpService.getResourceProfile(profileId);
+	}
+
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/common/ReadOnlyRepository.java

@@ -0,0 +1,22 @@
+package eu.dnetlib.openaire.common;
+
+import java.util.Optional;
+
+import org.springframework.data.domain.Page;
+import org.springframework.data.domain.Pageable;
+import org.springframework.data.repository.NoRepositoryBean;
+import org.springframework.data.repository.Repository;
+
+@NoRepositoryBean
+public interface ReadOnlyRepository<T, ID> extends Repository<T, ID> {
+
+	Optional<T> findById(ID id);
+
+	boolean existsById(ID id);
+
+	Page<T> findAll(Pageable pageable);
+
+	Iterable<T> findAll();
+
+	long count();
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityApiCore.java

@@ -1,368 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import static eu.dnetlib.openaire.community.CommunityConstants.CCONTENTPROVIDER_NAME;
-import static eu.dnetlib.openaire.community.CommunityConstants.CCONTENTPROVIDER_OFFICIALNAME;
-import static eu.dnetlib.openaire.community.CommunityConstants.CCONTENTPROVIDER_SELCRITERIA;
-import static eu.dnetlib.openaire.community.CommunityConstants.CLABEL;
-import static eu.dnetlib.openaire.community.CommunityConstants.CONTENTPROVIDERS_ID_SUFFIX;
-import static eu.dnetlib.openaire.community.CommunityConstants.CORGANIZATION_LOGOURL;
-import static eu.dnetlib.openaire.community.CommunityConstants.CORGANIZATION_NAME;
-import static eu.dnetlib.openaire.community.CommunityConstants.CORGANIZATION_WEBSITEURL;
-import static eu.dnetlib.openaire.community.CommunityConstants.CPROFILE_SUBJECT;
-import static eu.dnetlib.openaire.community.CommunityConstants.CPROJECT_ACRONYM;
-import static eu.dnetlib.openaire.community.CommunityConstants.CPROJECT_FULLNAME;
-import static eu.dnetlib.openaire.community.CommunityConstants.CPROJECT_FUNDER;
-import static eu.dnetlib.openaire.community.CommunityConstants.CPROJECT_NUMBER;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSUMMARY_DESCRIPTION;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSUMMARY_LOGOURL;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSUMMARY_NAME;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSUMMARY_STATUS;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSUMMARY_ZENODOC;
-import static eu.dnetlib.openaire.community.CommunityConstants.CSV_DELIMITER;
-import static eu.dnetlib.openaire.community.CommunityConstants.ID_SEPARATOR;
-import static eu.dnetlib.openaire.community.CommunityConstants.OPENAIRE_ID;
-import static eu.dnetlib.openaire.community.CommunityConstants.ORGANIZATION_ID_SUFFIX;
-import static eu.dnetlib.openaire.community.CommunityConstants.PROJECTS_ID_SUFFIX;
-import static eu.dnetlib.openaire.community.CommunityConstants.ZENODOCOMMUNITY_ID_SUFFIX;
-
-import java.util.Base64;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.TreeMap;
-import java.util.stream.Collectors;
-
-import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
-import org.springframework.cache.annotation.CacheEvict;
-import org.springframework.stereotype.Component;
-
-import com.google.common.base.Functions;
-import com.google.common.base.Joiner;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
-import eu.dnetlib.openaire.common.ISClient;
-
-@Component
-@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
-public class CommunityApiCore {// implements CommunityClient{
-
-	private static final Log log = LogFactory.getLog(CommunityApiCore.class);
-
-	@Autowired
-	private CommunityClient cci;
-
-	@Autowired
-	private ISClient isClient;
-
-	@Autowired
-	private CommunityCommon cc;
-
-	public List<CommunitySummary> listCommunities() throws CommunityException {
-		return cc.listCommunities();
-
-	}
-
-	public CommunityDetails getCommunity(final String id) throws CommunityException, CommunityNotFoundException {
-		return cc.getCommunity(id);
-
-	}
-
-	public void setCommunity(final String id, final CommunityWritableProperties details) throws CommunityException, CommunityNotFoundException {
-
-		cc.getCommunity(id); // ensure the community exists.
-
-		if (details.getShortName() != null) {
-			isClient.updateContextAttribute(id, CLABEL, details.getShortName());
-
-		}
-		if (details.getName() != null) {
-			isClient.updateContextParam(id, CSUMMARY_NAME, details.getName());
-
-		}
-		if (details.getDescription() != null) {
-			isClient.updateContextParam(id, CSUMMARY_DESCRIPTION, details.getDescription());
-
-		}
-		if (details.getLogoUrl() != null) {
-			isClient.updateContextParam(id, CSUMMARY_LOGOURL, details.getLogoUrl());
-
-		}
-		if (details.getStatus() != null) {
-			isClient.updateContextParam(id, CSUMMARY_STATUS, details.getStatus().name());
-
-		}
-		if (details.getSubjects() != null) {
-			isClient.updateContextParam(id, CPROFILE_SUBJECT, Joiner.on(CSV_DELIMITER).join(details.getSubjects()));
-
-		}
-		if (details.getMainZenodoCommunity() != null) {
-			isClient.updateContextParam(id, CSUMMARY_ZENODOC, details.getMainZenodoCommunity());
-		}
-
-		cc.updateCommunity(id, details);
-	}
-
-	public List<CommunityProject> getCommunityProjects(final String id) throws CommunityException, CommunityNotFoundException {
-		cc.getCommunity(id); // ensure the community exists.
-		return cc.getCommunityInfo(id, PROJECTS_ID_SUFFIX, c -> CommunityMappingUtils.asCommunityProject(id, c));
-	}
-
-	public CommunityProject addCommunityProject(final String id, final CommunityProject project) throws CommunityException, CommunityNotFoundException {
-		if (!StringUtils.equalsIgnoreCase(id, project.getCommunityId())) {
-			throw new CommunityException("parameters 'id' and project.communityId must be coherent");
-		}
-
-		final TreeMap<Integer, CommunityProject> projects = getCommunityProjectMap(id);
-		final String project_id = project.getId();
-
-		if (project_id != null && projects.keySet().contains(Integer.valueOf(project_id))) {
-			if (project.getName() != null) {
-				isClient.updateConceptParam(id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project_id, CPROJECT_FULLNAME, project.getName());
-
-			}
-			if (project.getAcronym() != null) {
-				isClient.updateConceptParam(id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project_id, CPROJECT_ACRONYM, project.getAcronym());
-
-			}
-			if (project.getOpenaireId() != null) {
-				isClient.updateConceptParam(id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project_id, OPENAIRE_ID, project.getOpenaireId());
-
-			}
-			if (project.getFunder() != null) {
-				isClient.updateConceptParam(id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project_id, CPROJECT_FUNDER, project.getFunder());
-
-			}
-			if (project.getGrantId() != null) {
-				isClient.updateConceptParam(id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project_id, CPROJECT_NUMBER, project.getGrantId());
-
-			}
-		} else {
-			project.setId(nextId(projects != null && !projects.isEmpty() ? projects.lastKey() : 0));
-
-			isClient.addConcept(id, id + PROJECTS_ID_SUFFIX, CommunityMappingUtils.asProjectXML(id, project));
-
-		}
-		cc.updateProject(id, project);
-		return project;
-	}
-
-	private String nextId(final Integer id) {
-		return String.valueOf(id + 1);
-	}
-
-	public void removeCommunityProject(final String id, final Integer projectId) throws CommunityException, CommunityNotFoundException {
-		final Map<Integer, CommunityProject> projects = getCommunityProjectMap(id);
-		if (!projects.containsKey(projectId)) {
-			throw new CommunityNotFoundException(String.format("project '%s' doesn't exist within context '%s'", projectId, id));
-		}
-		isClient.removeConcept(id, id + PROJECTS_ID_SUFFIX, id + PROJECTS_ID_SUFFIX + ID_SEPARATOR + projectId);
-		cc.removeFromCategory(id, PROJECTS_ID_SUFFIX, String.valueOf(projectId));
-	}
-
-	public List<CommunityContentprovider> getCommunityContentproviders(final String id) throws CommunityException, CommunityNotFoundException {
-		cc.getCommunity(id); // ensure the community exists.
-		return cc.getCommunityInfo(id, CONTENTPROVIDERS_ID_SUFFIX, c -> CommunityMappingUtils.asCommunityDataprovider(id, c));
-	}
-
-	public CommunityContentprovider addCommunityContentprovider(final String id, final CommunityContentprovider cp)
-		throws CommunityException, CommunityNotFoundException {
-		log.info("content provider to add " + cp.toString());
-		if (!StringUtils.equalsIgnoreCase(id, cp.getCommunityId())) { throw new CommunityException("parameters 'id' and cp.communityId must be coherent"); }
-
-		final TreeMap<Integer, CommunityContentprovider> cps = getCommunityContentproviderMap(id);
-		final String concept_id = cp.getId();
-		if (concept_id != null && cps.keySet().contains(Integer.valueOf(concept_id))) {
-			if (cp.getName() != null) {
-				isClient.updateConceptParam(id + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + concept_id, CCONTENTPROVIDER_NAME, cp.getName());
-			}
-			if (cp.getOfficialname() != null) {
-				isClient.updateConceptParam(id + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + concept_id, CCONTENTPROVIDER_OFFICIALNAME, cp.getOfficialname());
-			}
-			if (cp.getOpenaireId() != null) {
-				isClient.updateConceptParam(id + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + concept_id, OPENAIRE_ID, cp.getOpenaireId());
-			}
-			if (cp.getSelectioncriteria() != null) {
-				isClient.updateConceptParamNoEscape(id + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + concept_id, CCONTENTPROVIDER_SELCRITERIA, cp.toXML());
-			}
-		} else {
-			log.info("adding new concept for community " + id);
-			cp.setId(nextId(!cps.isEmpty() ? cps.lastKey() : 0));
-
-			isClient.addConcept(id, id + CONTENTPROVIDERS_ID_SUFFIX, CommunityMappingUtils.asContentProviderXML(id, cp));
-		}
-
-		cc.updateDatasource(id, cp);
-		return cp;
-	}
-
-	public void removeCommunityContentProvider(final String id, final Integer contentproviderId) throws CommunityException, CommunityNotFoundException {
-		final Map<Integer, CommunityContentprovider> providers = getCommunityContentproviderMap(id);
-		if (!providers.containsKey(contentproviderId)) {
-			throw new CommunityNotFoundException(String.format("content provider '%s' doesn't exist within context '%s'", contentproviderId, id));
-		}
-		isClient.removeConcept(id, id + CONTENTPROVIDERS_ID_SUFFIX, id + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + contentproviderId);
-		cc.removeFromCategory(id, CONTENTPROVIDERS_ID_SUFFIX, String.valueOf(contentproviderId));
-	}
-
-	public void removeCommunityOrganization(final String id, final Integer organizationId) throws CommunityException, CommunityNotFoundException {
-		final Map<Integer, CommunityOrganization> organizations = getCommunityOrganizationMap(id);
-		if (!organizations.containsKey(organizationId)) {
-			throw new CommunityNotFoundException(String.format("organization '%s' doesn't exist within context '%s'", organizationId, id));
-		}
-		isClient.removeConcept(id, id + ORGANIZATION_ID_SUFFIX, id + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organizationId);
-		cc.removeFromCategory(id, ORGANIZATION_ID_SUFFIX, String.valueOf(organizationId));
-	}
-
-	public List<CommunityZenodoCommunity> getCommunityZenodoCommunities(final String id) throws CommunityException, CommunityNotFoundException {
-
-		return cc.getCommunityZenodoCommunities(id);
-	}
-
-	public List<CommunityOrganization> getCommunityOrganizations(final String id) throws CommunityException, CommunityNotFoundException {
-		cc.getCommunity(id);
-		return cc.getCommunityInfo(id, ORGANIZATION_ID_SUFFIX, c -> CommunityMappingUtils.asCommunityOrganization(id, c));
-	}
-
-	public CommunityDetails addCommunitySubjects(final String id, final List<String> subjects) throws CommunityException, CommunityNotFoundException {
-
-		final CommunityDetails cd = new CommunityDetails();
-
-		final Set<String> current = Sets.newHashSet(cc.getCommunity(id).getSubjects());
-
-		current.addAll(subjects);
-
-		cd.setSubjects(Lists.newArrayList(current));
-
-		setCommunity(id, CommunityWritableProperties.fromDetails(cd));
-
-		return cd;
-	}
-
-	public CommunityDetails removeCommunitySubjects(final String id, final List<String> subjects) throws CommunityException, CommunityNotFoundException {
-
-		final CommunityDetails cd = new CommunityDetails();
-
-		final Set<String> current = Sets.newHashSet(cc.getCommunity(id).getSubjects());
-
-		current.removeAll(subjects);
-
-		cd.setSubjects(Lists.newArrayList(current));
-
-		setCommunity(id, CommunityWritableProperties.fromDetails(cd));
-
-		return cd;
-	}
-
-	@CacheEvict(value = "community-cache", allEntries = true)
-	public void removeCommunityZenodoCommunity(final String id, final Integer zenodoCommId) throws CommunityException, CommunityNotFoundException {
-
-		final Map<Integer, CommunityZenodoCommunity> zcomms = getZenodoCommunityMap(id);
-		if (!zcomms.containsKey(zenodoCommId)) {
-			throw new CommunityNotFoundException(String.format("Zenodo community '%s' doesn't exist within context '%s'", zenodoCommId, id));
-		}
-		isClient.removeConcept(id, id + ZENODOCOMMUNITY_ID_SUFFIX, id + ZENODOCOMMUNITY_ID_SUFFIX + ID_SEPARATOR + zenodoCommId);
-
-		cc.removeFromCategory(id, ZENODOCOMMUNITY_ID_SUFFIX, String.valueOf(zenodoCommId));
-
-	}
-
-	@CacheEvict(value = "community-cache", allEntries = true)
-	public CommunityZenodoCommunity addCommunityZenodoCommunity(final String id, final CommunityZenodoCommunity zc)
-		throws CommunityException, CommunityNotFoundException {
-		if (!StringUtils.equalsIgnoreCase(id, zc.getCommunityId())) { throw new CommunityException("parameters 'id' and zc.communityId must be coherent"); }
-		if (!StringUtils.isNotBlank(zc.getZenodoid())) { throw new CommunityException("parameter zenodoid cannot be null or empty"); }
-		final TreeMap<Integer, CommunityZenodoCommunity> zcs = getZenodoCommunityMap(id);
-
-		for (final CommunityZenodoCommunity czc : zcs.values()) {
-			if (czc.getZenodoid().equals(zc.getZenodoid())) { throw new CommunityException("Zenodo community already associated to the RCD"); }
-		}
-
-		zc.setId(nextId(!zcs.isEmpty() ? zcs.lastKey() : 0));
-
-		isClient.addConcept(id, id + ZENODOCOMMUNITY_ID_SUFFIX, CommunityMappingUtils.asZenodoCommunityXML(id, zc));
-		cc.updateZenodoCommunity(id, zc);
-
-		return zc;
-	}
-
-	public CommunityOpenAIRECommunities getOpenAIRECommunities(final String zenodoId) throws CommunityException, CommunityNotFoundException {
-
-		if (cci.getInverseZenodoCommunityMap().containsKey(zenodoId)) {
-			return new CommunityOpenAIRECommunities().setZenodoid(zenodoId)
-				.setOpenAirecommunitylist(cci.getInverseZenodoCommunityMap().get(zenodoId).stream().collect(Collectors.toList()));
-		}
-		return new CommunityOpenAIRECommunities();
-
-	}
-
-	// HELPERS
-
-	private TreeMap<Integer, CommunityProject> getCommunityProjectMap(final String id) throws CommunityException, CommunityNotFoundException {
-		return getCommunityProjects(id).stream()
-			.collect(Collectors.toMap(p -> Integer.valueOf(p.getId()), Functions.identity(), (p1, p2) -> {
-				log.warn(String.format("duplicate project found: '%s'", p1.getId()));
-				return p2;
-			}, TreeMap::new));
-	}
-
-	private TreeMap<Integer, CommunityContentprovider> getCommunityContentproviderMap(final String id) throws CommunityException, CommunityNotFoundException {
-		log.info("getting community content provider map");
-		return getCommunityContentproviders(id).stream()
-			.collect(Collectors.toMap(cp -> Integer.valueOf(cp.getId()), Functions.identity(), (cp1, cp2) -> {
-				log.warn(String.format("duplicate content provider found: '%s'", cp1.getId()));
-				return cp2;
-			}, TreeMap::new));
-	}
-
-	private TreeMap<Integer, CommunityZenodoCommunity> getZenodoCommunityMap(final String id) throws CommunityException, CommunityNotFoundException {
-		return getCommunityZenodoCommunities(id).stream()
-			.collect(Collectors.toMap(cp -> Integer.valueOf(cp.getId()), Functions.identity(), (cp1, cp2) -> {
-				log.warn(String.format("duplicate Zenodo community found: '%s'", cp1.getId()));
-				return cp2;
-			}, TreeMap::new));
-	}
-
-	private TreeMap<Integer, CommunityOrganization> getCommunityOrganizationMap(final String id) throws CommunityException, CommunityNotFoundException {
-		return getCommunityOrganizations(id).stream()
-			.collect(Collectors.toMap(o -> Integer.valueOf(o.getId()), Functions.identity(), (o1, o2) -> {
-				log.warn(String.format("duplicate content provider found: '%s'", o1.getId()));
-				return o2;
-			}, TreeMap::new));
-	}
-
-	public CommunityOrganization addCommunityOrganization(final String id, final CommunityOrganization organization)
-		throws CommunityException, CommunityNotFoundException {
-		if (!StringUtils.equalsIgnoreCase(id, organization.getCommunityId())) {
-			throw new CommunityException("parameters 'id' and organization.communityId must be coherent");
-		}
-
-		final TreeMap<Integer, CommunityOrganization> cps = getCommunityOrganizationMap(id);
-
-		final String organization_id = organization.getId();
-		if (organization_id != null && cps.keySet().contains(Integer.valueOf(organization_id))) {
-			if (organization.getName() != null) {
-				isClient.updateConceptParam(id + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization_id, CORGANIZATION_NAME, organization.getName());
-			}
-			if (organization.getLogo_url() != null) {
-				isClient.updateConceptParam(id + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization_id, CORGANIZATION_LOGOURL, Base64.getEncoder()
-					.encodeToString(organization.getLogo_url().getBytes()));
-			}
-			if (organization.getWebsite_url() != null) {
-				isClient.updateConceptParam(id + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization_id, CORGANIZATION_WEBSITEURL, Base64.getEncoder()
-					.encodeToString(organization.getWebsite_url().getBytes()));
-			}
-		} else {
-			organization.setId(nextId(!cps.isEmpty() ? cps.lastKey() : 0));
-			isClient.addConcept(id, id + ORGANIZATION_ID_SUFFIX, CommunityMappingUtils.asOrganizationXML(id, organization));
-		}
-
-		cc.updateOrganization(id, organization);
-		return organization;
-	}
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityClient.java

@@ -1,12 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.util.Map;
-import java.util.Set;
-
-public interface CommunityClient {
-
-    Map<String, Set<String>> getInverseZenodoCommunityMap() throws CommunityException, CommunityNotFoundException;
-
-    void dropCache();
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityClientImpl.java

@@ -1,58 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.cache.annotation.CacheEvict;
-import org.springframework.cache.annotation.Cacheable;
-import org.springframework.scheduling.annotation.Scheduled;
-import org.springframework.stereotype.Component;
-
-import java.util.*;
-
-@Component
-public class CommunityClientImpl implements CommunityClient {
-
-    private static final Log log = LogFactory.getLog(CommunityClient.class);
-
-    @Autowired
-    private CommunityCommon communityCommon;
-
-    @Override
-    @Cacheable("community-cache")
-    public Map<String, Set<String>> getInverseZenodoCommunityMap () throws CommunityException, CommunityNotFoundException {
-        log.info("Creating the data structure. Not using cache");
-        final Map<String, Set<String>> inverseListMap = new HashMap<>();
-
-        final List<CommunitySummary> communityList = communityCommon.listCommunities();
-
-        for(CommunitySummary cs :communityList){
-            final String communityId =  cs.getId();
-            List<CommunityZenodoCommunity> czc = communityCommon.getCommunityZenodoCommunities(communityId);
-            for(CommunityZenodoCommunity zc:czc){
-                final String zenodoId = zc.getZenodoid();
-                if(!inverseListMap.containsKey(zenodoId)) {
-                    inverseListMap.put(zc.getZenodoid(),new HashSet<>());
-                }
-                inverseListMap.get(zc.getZenodoid()).add(communityId);
-            }
-            final String zenodoMainCommunity = communityCommon.getCommunity(communityId).getZenodoCommunity();
-            if(!inverseListMap.containsKey(zenodoMainCommunity)) {
-                inverseListMap.put(zenodoMainCommunity,new HashSet<>());
-            }
-
-            inverseListMap.get(zenodoMainCommunity).add(communityId);
-        }
-        return inverseListMap;
-    }
-
-
-
-    @Override
-    @CacheEvict(cacheNames = { "community-cache", "context-cache-community"}, allEntries = true)
-    @Scheduled(fixedDelayString = "${openaire.exporter.cache.ttl}")
-    public void dropCache(){
-        log.debug("dropped community cache");
-    }
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityCommon.java

@@ -1,438 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import com.google.common.base.Joiner;
-import com.google.common.collect.Lists;
-import eu.dnetlib.openaire.common.ISClient;
-import eu.dnetlib.openaire.context.Category;
-import eu.dnetlib.openaire.context.Concept;
-import eu.dnetlib.openaire.context.Context;
-import eu.dnetlib.openaire.context.Param;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
-
-import java.io.IOException;
-import java.util.*;
-import java.util.function.Function;
-import java.util.stream.Collectors;
-
-import static eu.dnetlib.openaire.community.CommunityConstants.*;
-
-@Component
-public class CommunityCommon {
-
-    @Autowired
-    private ISClient isClient;
-
-    public Map<String, Context> getContextMap() throws CommunityException {
-        try {
-            return isClient.getCommunityContextMap();
-        } catch (IOException e) {
-            throw new CommunityException(e);
-        }
-    }
-
-    public List<CommunitySummary> listCommunities()  throws CommunityException {
-        return getContextMap().values().stream()
-                .filter(context -> !communityBlackList.contains(context.getId()))
-                .map(CommunityMappingUtils::asCommunitySummary)
-                .collect(Collectors.toList());
-    }
-
-
-    public <R> List<R> getCommunityInfo(final String id, final String idSuffix, final Function<Concept, R> mapping) throws CommunityException {
-        final Map<String, Context> contextMap = getContextMap();
-        final Context context = contextMap.get(id);
-        if (context != null) {
-            final Map<String, Category> categories = context.getCategories();
-            final Category category = categories.get(id + idSuffix);
-            if (category != null) {
-                return category.getConcepts().stream()
-                        .map(mapping)
-                        .collect(Collectors.toList());
-            }
-        }
-        return Lists.newArrayList();
-    }
-
-    public CommunityDetails getCommunity(final String id) throws CommunityException, CommunityNotFoundException {
-        final Context context = getContextMap().get(id);
-        if (context == null || CommunityConstants.communityBlackList.contains(id)) {
-            throw new CommunityNotFoundException(String.format("community '%s' does not exist", id));
-        }
-        return CommunityMappingUtils.asCommunityProfile(context);
-    }
-
-    public List<CommunityZenodoCommunity> getCommunityZenodoCommunities(final String id) throws CommunityException, CommunityNotFoundException {
-        getCommunity(id); // ensure the community exists.
-        return getCommunityInfo(id, ZENODOCOMMUNITY_ID_SUFFIX, c -> CommunityMappingUtils.asCommunityZenodoCommunity(id, c));
-    }
-
-
-    public void updateProject(String communityId, CommunityProject project) throws CommunityException {
-        final Context context = getContextMap().get(communityId);
-        Category prj = context.getCategories().get(communityId + PROJECTS_ID_SUFFIX);
-        if (prj.getConcepts().stream().map(c -> c.getId()).collect(Collectors.toList())
-                .contains(communityId + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project.getId())){
-            prj.getConcepts().forEach(concept -> {
-                if (concept.getId().equals(communityId + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project.getId())) {
-                    if (project.getName() != null) {
-
-                        concept.getParams().replace(CPROJECT_FULLNAME, Arrays.asList(new Param()
-                                .setName(CPROJECT_FULLNAME).setValue(project.getName())));
-                    }
-                    if (project.getAcronym() != null) {
-                        if(concept.getParams().keySet().contains(CPROJECT_ACRONYM)){
-                            concept.getParams().replace(CPROJECT_ACRONYM, Arrays.asList(new Param()
-                                    .setName(CPROJECT_ACRONYM).setValue(project.getAcronym())));
-                        }
-                        else{
-                            concept.getParams().put(CPROJECT_ACRONYM, Arrays.asList(new Param()
-                                    .setName(CPROJECT_ACRONYM).setValue(project.getAcronym())));
-                        }
-
-                    }
-                    if (project.getOpenaireId() != null) {
-                        if(concept.getParams().keySet().contains(OPENAIRE_ID)){
-                            concept.getParams().replace(OPENAIRE_ID, Arrays.asList(new Param()
-                                    .setName(OPENAIRE_ID).setValue(project.getOpenaireId())));
-                        }
-                        else{
-                            concept.getParams().put(OPENAIRE_ID, Arrays.asList(new Param()
-                            .setName(OPENAIRE_ID).setValue(project.getOpenaireId())));
-                        }
-
-                    }
-                    if (project.getFunder() != null) {
-                        concept.getParams().replace(CPROJECT_FUNDER, Arrays.asList(new Param()
-                                .setName(CPROJECT_FUNDER).setValue(project.getFunder())));
-                    }
-                    if (project.getGrantId() != null) {
-                        concept.getParams().replace(CPROJECT_NUMBER, Arrays.asList(new Param()
-                                .setName(CPROJECT_NUMBER).setValue(project.getGrantId())));
-
-                    }
-                }
-            });
-        }
-        else{
-            Concept concept = new Concept();
-            concept.setId(communityId + PROJECTS_ID_SUFFIX + ID_SEPARATOR + project.getId());
-            concept.setClaim(false);
-            if(project.getAcronym() != null)
-                concept.setLabel(project.getAcronym());
-            else
-                concept.setLabel("");
-
-             Map<String, List<Param>> params = new TreeMap<>();
-
-             if(project.getAcronym() != null){
-                 params.put(CPROJECT_ACRONYM, Arrays.asList(new Param().setName(CPROJECT_ACRONYM)
-                         .setValue(project.getAcronym())));
-             }
-
-             if (project.getName() != null){
-                 params.put(CPROJECT_FULLNAME, Arrays.asList(new Param()
-                         .setName(CPROJECT_FULLNAME)
-                         .setValue(project.getName())
-                 ));
-             }
-
-             if (project.getOpenaireId() != null){
-                 params.put(OPENAIRE_ID, Arrays.asList(new Param()
-                         .setName(OPENAIRE_ID)
-                         .setValue(project.getOpenaireId())
-                 ));
-             }
-
-             if(project.getFunder() != null){
-                 params.put(CPROJECT_FUNDER, Arrays.asList(new Param()
-                         .setName(CPROJECT_FUNDER)
-                         .setValue(project.getFunder())
-                 ));
-             }
-
-             if (project.getGrantId()!=null){
-                 params.put(CPROJECT_NUMBER, Arrays.asList(new Param()
-                         .setName(CPROJECT_NUMBER)
-                         .setValue(project.getGrantId())
-                 ));
-             }
-
-            concept.setParams(params);
-            prj.getConcepts().add(concept);
-
-
-        }
-
-    }
-
-    public void updateCommunity(String id, CommunityWritableProperties community) throws CommunityException {
-        final Context context = getContextMap().get(id);
-
-        if(community.getShortName() != null) {
-            context.setLabel(community.getShortName());
-        }
-
-        if (community.getName() != null){
-            context.getParams().replace(CSUMMARY_NAME, Arrays.asList(new Param()
-            .setValue(community.getName()).setName(CSUMMARY_NAME)));
-        }
-        if(community.getDescription() != null) {
-            context.getParams()
-                    .replace(CSUMMARY_DESCRIPTION, Arrays.asList(new Param()
-                            .setName(CSUMMARY_DESCRIPTION).setValue(community.getDescription())));
-        }
-        if(community.getLogoUrl() != null){
-            context.getParams()
-                    .replace(CSUMMARY_LOGOURL, Arrays.asList(new Param()
-                    .setName(CSUMMARY_LOGOURL).setValue(community.getLogoUrl())));
-
-        }
-        if (community.getStatus() != null) {
-            context.getParams()
-                    .replace(CSUMMARY_STATUS, Arrays.asList(new Param()
-                    .setName(CSUMMARY_STATUS).setValue(community.getStatus().name())));
-        }
-        if (community.getSubjects() != null) {
-            context.getParams()
-                    .replace(CPROFILE_SUBJECT, Arrays.asList(new Param().setName(CPROFILE_SUBJECT)
-                            .setValue(Joiner.on(CSV_DELIMITER)
-                                    .join(community.getSubjects()))));
-        }
-        if(community.getMainZenodoCommunity() != null){
-            context.getParams()
-                    .replace(CSUMMARY_ZENODOC, Arrays.asList(new Param()
-                            .setName(CSUMMARY_ZENODOC).setValue(community.getMainZenodoCommunity())));
-
-        }
-
-    }
-
-
-    public void removeFromCategory(String communityId, String category, String conceptId) throws CommunityException {
-        Map<String, Context> cmap = getContextMap();
-        Context context = cmap.get(communityId);
-        Map<String, Category> cat = context.getCategories();
-
-        List<Concept> concepts = cat.get(communityId + category).getConcepts()
-                .stream().filter(c -> !c.getId().equals(communityId + category + ID_SEPARATOR + conceptId)).collect(Collectors.toList());
-
-         cat.get(communityId + category).setConcepts(concepts);
-    }
-
-    public void updateDatasource(String communityId, CommunityContentprovider cp) throws CommunityException {
-        final Context context = getContextMap().get(communityId);
-        Category dts = context.getCategories().get(communityId + CONTENTPROVIDERS_ID_SUFFIX);
-        if (dts.getConcepts().stream().map(c -> c.getId()).collect(Collectors.toList())
-                .contains(communityId + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + cp.getId())){
-            dts.getConcepts().forEach(concept -> {
-                if (concept.getId().equals(communityId + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + cp.getId())) {
-
-
-                    if (cp.getName() != null) {
-                        if(concept.getParams().keySet().contains(CCONTENTPROVIDER_NAME)){
-                            concept.getParams().replace(CCONTENTPROVIDER_NAME, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_NAME).setValue(cp.getName())));
-                        }
-                        else{
-                            concept.getParams().put(CCONTENTPROVIDER_NAME, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_NAME).setValue(cp.getName())));
-                        }
-
-                    }
-                    if (cp.getOfficialname() != null) {
-                        if(concept.getParams().keySet().contains(CCONTENTPROVIDER_OFFICIALNAME)){
-                            concept.getParams().replace(CCONTENTPROVIDER_OFFICIALNAME, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_OFFICIALNAME).setValue(cp.getOfficialname())));
-                        }
-                        else{
-                            concept.getParams().put(CCONTENTPROVIDER_OFFICIALNAME, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_OFFICIALNAME).setValue(cp.getOfficialname())));
-                        }
-
-                    }
-
-
-                    if (cp.getOpenaireId() != null) {
-                        if(concept.getParams().keySet().contains(OPENAIRE_ID)){
-                            concept.getParams().replace(OPENAIRE_ID, Arrays.asList(new Param()
-                                    .setName(OPENAIRE_ID).setValue(cp.getOpenaireId())));
-                        }
-                        else{
-                            concept.getParams().put(OPENAIRE_ID, Arrays.asList(new Param()
-                                    .setName(OPENAIRE_ID).setValue(cp.getOpenaireId())));
-                        }
-
-                    }
-
-                    if (cp.getSelectioncriteria()  != null) {
-                        if(concept.getParams().keySet().contains(CCONTENTPROVIDER_SELCRITERIA)){
-                            concept.getParams().replace(CCONTENTPROVIDER_SELCRITERIA, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_SELCRITERIA).setValue(cp.toJson())));
-                        }
-                        else{
-                            concept.getParams().put(CCONTENTPROVIDER_SELCRITERIA, Arrays.asList(new Param()
-                                    .setName(CCONTENTPROVIDER_SELCRITERIA).setValue(cp.toJson())));
-                        }
-
-                    }
-                }
-            });
-        }
-        else{
-            Concept concept = new Concept();
-            concept.setId(communityId + CONTENTPROVIDERS_ID_SUFFIX + ID_SEPARATOR + cp.getId());
-            concept.setClaim(false);
-            concept.setLabel("");
-
-            Map<String, List<Param>> params = new TreeMap<>();
-
-            if (cp.getName() != null) {
-                params.put( CCONTENTPROVIDER_NAME, Arrays.asList(new Param().setValue(cp.getName()).setName(CCONTENTPROVIDER_NAME)));
-            }
-            if(cp.getOfficialname()!= null){
-                params.put( CCONTENTPROVIDER_OFFICIALNAME, Arrays.asList(new Param().setValue(cp.getOfficialname()).setName(CCONTENTPROVIDER_OFFICIALNAME)));
-            }
-            if (cp.getOpenaireId() != null){
-                params.put( OPENAIRE_ID, Arrays.asList(new Param().setValue(cp.getOpenaireId()).setName(OPENAIRE_ID)));
-            }
-            if(cp.getSelectioncriteria() != null){
-                params.put( CCONTENTPROVIDER_SELCRITERIA, Arrays.asList(new Param().setValue(cp.toJson()).setName(CCONTENTPROVIDER_SELCRITERIA)));
-
-            }
-
-            concept.setParams(params);
-            dts.getConcepts().add(concept);
-
-
-        }
-
-    }
-
-    public void updateOrganization(String communityId, CommunityOrganization organization) throws CommunityException {
-
-        final Context context = getContextMap().get(communityId);
-        Category orgs = context.getCategories().get(communityId + ORGANIZATION_ID_SUFFIX);
-        if (orgs.getConcepts().stream().map(c -> c.getId()).collect(Collectors.toList())
-                .contains(communityId + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization.getId())){
-            orgs.getConcepts().forEach(concept -> {
-                if (concept.getId().equals(communityId + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization.getId())) {
-
-
-                    if (organization.getName() != null) {
-                        if(concept.getParams().keySet().contains(CORGANIZATION_NAME)){
-                            concept.getParams().replace(CORGANIZATION_NAME, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_NAME).setValue(organization.getName())));
-                        }
-                        else{
-                            concept.getParams().put(CORGANIZATION_NAME, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_NAME).setValue(organization.getName())));
-                        }
-
-                    }
-                    if (organization.getLogo_url() != null) {
-                        if(concept.getParams().keySet().contains(CORGANIZATION_LOGOURL)){
-                            concept.getParams().replace(CORGANIZATION_LOGOURL, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_LOGOURL).setValue(Base64.getEncoder().encodeToString(organization.getLogo_url().getBytes()))));
-                        }
-                        else{
-                            concept.getParams().put(CORGANIZATION_LOGOURL, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_LOGOURL).setValue(Base64.getEncoder().encodeToString(organization.getLogo_url().getBytes()))));
-                        }
-
-                    }
-
-
-                    if (organization.getWebsite_url() != null) {
-                        if(concept.getParams().keySet().contains(CORGANIZATION_WEBSITEURL)){
-                            concept.getParams().replace(CORGANIZATION_WEBSITEURL, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_WEBSITEURL).setValue(Base64.getEncoder().encodeToString(organization.getWebsite_url().getBytes()))));
-                        }
-                        else{
-                            concept.getParams().put(CORGANIZATION_WEBSITEURL, Arrays.asList(new Param()
-                                    .setName(CORGANIZATION_WEBSITEURL).setValue(Base64.getEncoder().encodeToString(organization.getWebsite_url().getBytes()))));
-                        }
-
-                    }
-
-                    }
-
-            });
-        }
-        else{
-
-            Concept concept = new Concept();
-            concept.setId(communityId + ORGANIZATION_ID_SUFFIX + ID_SEPARATOR + organization.getId());
-            concept.setClaim(false);
-            concept.setLabel("");
-
-            Map<String, List<Param>> params = new TreeMap<>();
-
-            if (organization.getName() != null) {
-                params.put( CORGANIZATION_NAME, Arrays.asList(new Param().setValue(organization.getName()).setName(CORGANIZATION_NAME)));
-            }
-            if(organization.getLogo_url()!= null){
-
-                params.put( CORGANIZATION_LOGOURL, Arrays.asList(new Param().setValue(Base64.getEncoder().encodeToString(organization.getLogo_url().getBytes())).setName(CORGANIZATION_LOGOURL)));
-            }
-            if (organization.getWebsite_url() != null){
-                params.put( CORGANIZATION_WEBSITEURL, Arrays.asList(new Param().setValue(Base64.getEncoder().encodeToString(organization.getWebsite_url().getBytes())).setName(CORGANIZATION_WEBSITEURL)));
-            }
-
-
-            concept.setParams(params);
-            orgs.getConcepts().add(concept);
-
-
-        }
-
-    }
-
-    public void updateZenodoCommunity(String communityId, CommunityZenodoCommunity zc) throws CommunityException {
-        final Context context = getContextMap().get(communityId);
-        Category zcs = context.getCategories().get(communityId + ZENODOCOMMUNITY_ID_SUFFIX);
-        if (zcs.getConcepts().stream().map(c -> c.getId()).collect(Collectors.toList())
-                .contains(communityId + ZENODOCOMMUNITY_ID_SUFFIX + ID_SEPARATOR + zc.getId())){
-            zcs.getConcepts().forEach(concept -> {
-                if (concept.getId().equals(communityId + ZENODOCOMMUNITY_ID_SUFFIX + ID_SEPARATOR + zc.getId())) {
-
-
-                    if (zc.getZenodoid() != null) {
-                        if(concept.getParams().keySet().contains(CZENODOCOMMUNITY_ID)){
-                            concept.getParams().replace(CZENODOCOMMUNITY_ID, Arrays.asList(new Param()
-                                    .setName(CZENODOCOMMUNITY_ID).setValue(zc.getZenodoid())));
-                        }
-                        else{
-                            concept.getParams().put(CZENODOCOMMUNITY_ID, Arrays.asList(new Param()
-                                    .setName(CZENODOCOMMUNITY_ID).setValue(zc.getZenodoid())));
-                        }
-
-                    }
-
-                }
-
-            });
-        }
-        else{
-
-            Concept concept = new Concept();
-            concept.setId(communityId + ZENODOCOMMUNITY_ID_SUFFIX + ID_SEPARATOR + zc.getId());
-            concept.setClaim(false);
-
-
-            Map<String, List<Param>> params = new TreeMap<>();
-
-            if (zc.getZenodoid() != null) {
-                params.put( CZENODOCOMMUNITY_ID, Arrays.asList(new Param().setValue(zc.getZenodoid()).setName(CZENODOCOMMUNITY_ID)));
-                concept.setLabel(zc.getZenodoid());
-            }else{
-                concept.setLabel("");
-            }
-            concept.setParams(params);
-            zcs.getConcepts().add(concept);
-
-        }
-
-    }
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityConstants.java

@@ -1,58 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.util.Set;
-
-import com.google.common.collect.Sets;
-
-public class CommunityConstants {
-
-	public final static Set<String> communityBlackList = Sets.newHashSet("fet-fp7", "fet-h2020");
-
-	// common
-	public final static String OPENAIRE_ID = "openaireId";
-	public final static String PIPE_SEPARATOR = "||";
-	public final static String ID_SEPARATOR = "::";
-	public final static String CSV_DELIMITER = ",";
-	public final static String CLABEL = "label";
-
-	// id suffixes
-	public final static String PROJECTS_ID_SUFFIX = ID_SEPARATOR + "projects";
-	public final static String CONTENTPROVIDERS_ID_SUFFIX = ID_SEPARATOR + "contentproviders";
-	public final static String ZENODOCOMMUNITY_ID_SUFFIX = ID_SEPARATOR + "zenodocommunities";
-	public final static String ORGANIZATION_ID_SUFFIX = ID_SEPARATOR + "organizations";
-
-	// community summary
-	public final static String CSUMMARY_DESCRIPTION = "description";
-	public final static String CSUMMARY_LOGOURL = "logourl";
-	public final static String CSUMMARY_STATUS = "status";
-	public final static String CSUMMARY_NAME = "name";
-	public final static String CSUMMARY_MANAGER = "manager";
-	public final static String CSUMMARY_ZENODOC = "zenodoCommunity";
-
-	// community profile
-	public final static String CPROFILE_SUBJECT = "subject";
-	public final static String CPROFILE_CREATIONDATE = "creationdate";
-
-	// community project
-	public final static String CPROJECT_FUNDER = "funder";
-	public final static String CPROJECT_NUMBER = "CD_PROJECT_NUMBER";
-	public final static String CPROJECT_FULLNAME = "projectfullname";
-	public final static String CPROJECT_ACRONYM = "acronym";
-
-	// community content provider
-	public final static String CCONTENTPROVIDER_NAME = "name";
-	public final static String CCONTENTPROVIDER_OFFICIALNAME = "officialname";
-	public final static String CCONTENTPROVIDER_ENABLED = "enabled";
-	public final static String CCONTENTPROVIDERENABLED_DEFAULT = "true";
-	public final static String CCONTENTPROVIDER_SELCRITERIA = "selcriteria";
-
-	//community zenodo community
-	public final static String CZENODOCOMMUNITY_ID = "zenodoid";
-
-	//community organization
-	public final static String CORGANIZATION_NAME = "name";
-	public final static String CORGANIZATION_LOGOURL = "logourl";
-	public final static String CORGANIZATION_WEBSITEURL = "websiteurl";
-
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityDetails.java

@@ -1,55 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.util.Date;
-import java.util.List;
-
-import com.fasterxml.jackson.annotation.JsonAutoDetect;
-
-import io.swagger.v3.oas.annotations.media.Schema;
-
-@JsonAutoDetect
-public class CommunityDetails extends CommunitySummary {
-
-	@Schema(description = "date of creation for this community")
-	private Date creationDate;
-
-	@Schema(description = "date of the last update for this communityu")
-	private Date lastUpdateDate;
-
-	@Schema(description = "list of subjects (keywords) that characterise this community")
-	private List<String> subjects;
-
-	public CommunityDetails() {}
-
-	public CommunityDetails(final CommunitySummary summary) {
-		super(summary);
-	}
-
-	@Override
-	public Date getCreationDate() {
-		return creationDate;
-	}
-
-	@Override
-	public void setCreationDate(final Date creationDate) {
-		this.creationDate = creationDate;
-	}
-
-	public List<String> getSubjects() {
-		return subjects;
-	}
-
-	public void setSubjects(final List<String> subjects) {
-		this.subjects = subjects;
-	}
-
-	@Override
-	public Date getLastUpdateDate() {
-		return lastUpdateDate;
-	}
-
-	@Override
-	public void setLastUpdateDate(final Date lastUpdateDate) {
-		this.lastUpdateDate = lastUpdateDate;
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityException.java

@@ -1,25 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.io.IOException;
-
-import org.springframework.http.HttpStatus;
-import org.springframework.web.bind.annotation.ResponseBody;
-import org.springframework.web.bind.annotation.ResponseStatus;
-
-@ResponseBody
-@ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
-public class CommunityException extends Exception {
-
-	/**
-	 *
-	 */
-	private static final long serialVersionUID = -4961233580574761346L;
-
-	public CommunityException(final String message) {
-		super(message);
-	}
-
-	public CommunityException(final IOException e) {
-		super(e);
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityMappingUtils.java

@@ -1,231 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.text.ParseException;
-import java.util.*;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
-
-import com.google.common.escape.Escaper;
-import com.google.common.xml.XmlEscapers;
-import eu.dnetlib.openaire.community.selectioncriteria.SelectionCriteria;
-import eu.dnetlib.openaire.context.Concept;
-import eu.dnetlib.openaire.context.Context;
-import eu.dnetlib.openaire.context.Param;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-
-import static eu.dnetlib.openaire.common.Utils.escape;
-import static eu.dnetlib.openaire.community.CommunityConstants.*;
-
-public class CommunityMappingUtils {
-
-	private final static String pattern = "yyyy-MM-dd'T'hh:mm:ss";
-
-	private static final Log log = LogFactory.getLog(CommunityMappingUtils.class);
-
-	public static CommunitySummary asCommunitySummary(final Context c) {
-		final CommunitySummary summary = new CommunitySummary();
-
-		summary.setId(c.getId());
-		summary.setShortName(c.getLabel());
-		summary.setLastUpdateDate(c.getLastUpdateDate());
-		summary.setCreationDate(c.getCreationDate());
-		summary.setQueryId(c.getId() + PIPE_SEPARATOR + c.getLabel());
-		summary.setType(c.getType());
-
-		final Map<String, List<Param>> params = c.getParams();
-		if (params.containsKey(CSUMMARY_DESCRIPTION)) {
-			summary.setDescription(asCsv(params.get(CSUMMARY_DESCRIPTION)));
-		}
-		if (params.containsKey(CSUMMARY_LOGOURL)) {
-			summary.setLogoUrl(asCsv(params.get(CSUMMARY_LOGOURL)));
-		}
-		if (params.containsKey(CSUMMARY_STATUS)) {
-			summary.setStatus(CommunityStatus.valueOf(firstValue(params, CSUMMARY_STATUS)));
-		}
-		if (params.containsKey(CSUMMARY_NAME)) {
-			summary.setName(asCsv(params.get(CSUMMARY_NAME)));
-		}
-		if (params.containsKey(CSUMMARY_ZENODOC)) {
-			summary.setZenodoCommunity(asCsv(params.get(CSUMMARY_ZENODOC)));
-		}
-
-		return summary;
-	}
-
-	public static CommunityDetails asCommunityProfile(final Context c)  {
-
-		final CommunityDetails p = new CommunityDetails(asCommunitySummary(c));
-		p.setLastUpdateDate(c.getLastUpdateDate());
-		final Map<String, List<Param>> params = c.getParams();
-		if (params.containsKey(CPROFILE_SUBJECT)) {
-			p.setSubjects(splitValues(asValues(params.get(CPROFILE_SUBJECT)), CSV_DELIMITER));
-		}
-		if (params.containsKey(CPROFILE_CREATIONDATE)){
-			try {
-				p.setCreationDate(org.apache.commons.lang3.time.DateUtils.parseDate(asCsv(params.get(CPROFILE_CREATIONDATE)), pattern));
-			}catch(ParseException e) {
-				e.printStackTrace();
-			}
-		}
-
-		return p;
-	}
-
-	public static CommunityProject asCommunityProject(final String communityId, final Concept c) {
-
-		final Map<String, List<Param>> p = c.getParams();
-		final CommunityProject project = new CommunityProject();
-		project.setCommunityId(communityId);
-		project.setId(StringUtils.substringAfterLast(c.getId(), ID_SEPARATOR));
-		project.setOpenaireId(firstValue(p, OPENAIRE_ID));
-		project.setFunder(firstValue(p, CPROJECT_FUNDER));
-		project.setGrantId(firstValue(p, CPROJECT_NUMBER));
-		project.setName(firstValue(p, CPROJECT_FULLNAME));
-		project.setAcronym(firstValue(p, CPROJECT_ACRONYM));
-
-		return project;
-	}
-
-	public static CommunityContentprovider asCommunityDataprovider(final String communityId, final Concept c) {
-
-		final Map<String, List<Param>> p = c.getParams();
-		final CommunityContentprovider d = new CommunityContentprovider();
-		d.setCommunityId(communityId);
-		d.setId(StringUtils.substringAfterLast(c.getId(), ID_SEPARATOR));
-		d.setOpenaireId(firstValue(p, OPENAIRE_ID));
-		d.setName(firstValue(p, CCONTENTPROVIDER_NAME));
-		d.setOfficialname(firstValue(p, CCONTENTPROVIDER_OFFICIALNAME));
-		d.setSelectioncriteria(SelectionCriteria.fromJson(firstValue(p, CCONTENTPROVIDER_SELCRITERIA)));
-		return d;
-	}
-
-	public static CommunityZenodoCommunity asCommunityZenodoCommunity(final String communityId, final Concept c){
-		final CommunityZenodoCommunity z = new CommunityZenodoCommunity();
-		final Map<String, List<Param>> p = c.getParams();
-		z.setCommunityId(communityId);
-		z.setId(StringUtils.substringAfterLast(c.getId(), ID_SEPARATOR));
-		z.setZenodoid(firstValue(p,CZENODOCOMMUNITY_ID));
-		//z.setName(c.getLabel());
-		return z;
-	}
-
-
-	public static CommunityOrganization asCommunityOrganization(String id, Concept c) {
-		final Map<String, List<Param>> p = c.getParams();
-		final CommunityOrganization o = new CommunityOrganization();
-		o.setCommunityId(id);
-		o.setId(StringUtils.substringAfterLast(c.getId(), ID_SEPARATOR));
-		o.setName(firstValue(p,CORGANIZATION_NAME));
-		o.setLogo_url(getDecodedUrl(firstValue(p,CORGANIZATION_LOGOURL)));
-		o.setWebsite_url(getDecodedUrl(firstValue(p,CORGANIZATION_WEBSITEURL)));
-
-		return o;
-	}
-
-	private static String getDecodedUrl(final String encoded_url){
-		if(encoded_url == null){
-			return encoded_url;
-		}
-		return new String(Base64.getDecoder().decode(encoded_url));
-	}
-
-
-	private static List<String> splitValues(final Stream<String> stream, final String separator) {
-		return stream.map(s -> s.split(separator))
-				.map(Arrays::asList)
-				.flatMap(List::stream)
-				.filter(StringUtils::isNotBlank)
-				.map(StringUtils::trim)
-				.collect(Collectors.toList());
-	}
-
-	private static String firstValue(final Map<String, List<Param>> p, final String paramName) {
-		return asValues(p.get(paramName)).findFirst().orElse(null);
-	}
-
-	private static String asCsv(final List<Param> params) {
-		return asValues(params)
-				.collect(Collectors.joining(CSV_DELIMITER));
-	}
-
-	private static Stream<String> asValues(final List<Param> params) {
-		return params == null ? Stream.empty() : params.stream()
-				.map(Param::getValue)
-				.map(StringUtils::trim)
-				.distinct();
-	}
-
-	public static String asProjectXML(final String contextId, final CommunityProject project) {
-		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
-		final StringBuilder sb = new StringBuilder();
-		sb.append(
-				String.format(
-						"<concept claim='false' id='%s%s%s%s' label='%s'>\n",
-						escape(esc, contextId), PROJECTS_ID_SUFFIX, ID_SEPARATOR, escape(esc, String.valueOf(project.getId())), escape(esc, project.getAcronym())));
-		sb.append(paramXML(CPROJECT_FULLNAME, project.getName()));
-		sb.append(paramXML(CPROJECT_ACRONYM, project.getAcronym()));
-		sb.append(paramXML(CPROJECT_NUMBER, project.getGrantId()));
-		sb.append(paramXML(CPROJECT_FUNDER, project.getFunder()));
-		sb.append(paramXML(OPENAIRE_ID, project.getOpenaireId()));
-		sb.append("</concept>\n");
-		return sb.toString();
-	}
-
-	public static String asContentProviderXML(final String contextId, final CommunityContentprovider ccp) {
-		log.info("creating the XML for the content provider");
-		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
-		final StringBuilder sb = new StringBuilder();
-		sb.append(
-				String.format(
-						"<concept claim='false' id='%s%s%s%s' label='%s'>\n",
-						escape(esc, contextId), CONTENTPROVIDERS_ID_SUFFIX, ID_SEPARATOR, escape(esc, String.valueOf(ccp.getId())), escape(esc, ccp.getName())));
-		sb.append(paramXML(OPENAIRE_ID, ccp.getOpenaireId()));
-		sb.append(paramXML(CCONTENTPROVIDER_NAME, ccp.getName()));
-		sb.append(paramXML(CCONTENTPROVIDER_OFFICIALNAME, ccp.getOfficialname()));
-		sb.append(paramXML(CCONTENTPROVIDER_ENABLED,CCONTENTPROVIDERENABLED_DEFAULT));
-		sb.append(paramXMLNoEscape(CCONTENTPROVIDER_SELCRITERIA, ccp.toXML()));
-		sb.append("</concept>\n");
-		log.info(sb.toString());
-		return sb.toString();
-	}
-
-	public static String asZenodoCommunityXML(final String contextId, final CommunityZenodoCommunity zc) {
-		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
-		final StringBuilder sb = new StringBuilder();
-		sb.append(
-				String.format(
-						"<concept claim='false' id='%s%s%s%s' label='%s'>\n",
-						escape(esc, contextId), ZENODOCOMMUNITY_ID_SUFFIX, ID_SEPARATOR, escape(esc, String.valueOf(zc.getId())), escape(esc, zc.getZenodoid())));
-
-		sb.append(paramXML(CZENODOCOMMUNITY_ID, zc.getZenodoid()));
-		sb.append("</concept>\n");
-		return sb.toString();
-	}
-
-
-	public static String asOrganizationXML(final String contextId, CommunityOrganization organization) {
-		final Escaper esc = XmlEscapers.xmlAttributeEscaper();
-		final StringBuilder sb = new StringBuilder();
-		sb.append(
-				String.format(
-						"<concept claim='false' id='%s%s%s%s' label='%s'>\n",
-						escape(esc, contextId), ORGANIZATION_ID_SUFFIX, ID_SEPARATOR, escape(esc, String.valueOf(organization.getId())), escape(esc, organization.getName())));
-		sb.append(paramXML(CORGANIZATION_NAME, organization.getName()));
-		sb.append(paramXML(CORGANIZATION_LOGOURL, Base64.getEncoder().encodeToString(organization.getLogo_url().getBytes())));
-		sb.append(paramXML(CORGANIZATION_WEBSITEURL,Base64.getEncoder().encodeToString(organization.getWebsite_url().getBytes())));
-		sb.append("</concept>\n");
-		return sb.toString();
-	}
-
-
-	private static String paramXML(final String paramName, final String value) {
-		return String.format("<param name='%s'>%s</param>\n", paramName, escape(XmlEscapers.xmlContentEscaper(), value));
-	}
-
-	private static String paramXMLNoEscape(final String paramName, final String value) {
-		return String.format("<param name='%s'>%s</param>\n", paramName, value);
-	}
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityNotFoundException.java

@@ -1,24 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import org.springframework.http.HttpStatus;
-import org.springframework.web.bind.annotation.ResponseBody;
-import org.springframework.web.bind.annotation.ResponseStatus;
-
-@ResponseBody
-@ResponseStatus(value = HttpStatus.NOT_FOUND)
-public class CommunityNotFoundException extends Exception {
-
-	/**
-	 *
-	 */
-	private static final long serialVersionUID = -5605421323034135778L;
-
-	public CommunityNotFoundException(final String msg) {
-		super(msg);
-	}
-
-	public CommunityNotFoundException(final Exception e) {
-		super(e);
-	}
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityService.java

@@ -0,0 +1,479 @@
+package eu.dnetlib.openaire.community;
+
+import java.time.LocalDateTime;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import javax.persistence.criteria.Predicate;
+import javax.transaction.Transactional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.domain.Page;
+import org.springframework.data.domain.PageRequest;
+import org.springframework.data.domain.Sort;
+import org.springframework.data.jpa.domain.Specification;
+import org.springframework.stereotype.Service;
+
+import eu.dnetlib.openaire.community.model.DbCommunity;
+import eu.dnetlib.openaire.community.model.DbDatasource;
+import eu.dnetlib.openaire.community.model.DbDatasourcePK;
+import eu.dnetlib.openaire.community.model.DbOrganization;
+import eu.dnetlib.openaire.community.model.DbProject;
+import eu.dnetlib.openaire.community.model.DbProjectPK;
+import eu.dnetlib.openaire.community.model.DbSubCommunity;
+import eu.dnetlib.openaire.community.model.DbSupportOrg;
+import eu.dnetlib.openaire.community.model.DbSupportOrgPK;
+import eu.dnetlib.openaire.community.repository.DbCommunityRepository;
+import eu.dnetlib.openaire.community.repository.DbDatasourceRepository;
+import eu.dnetlib.openaire.community.repository.DbOrganizationRepository;
+import eu.dnetlib.openaire.community.repository.DbProjectRepository;
+import eu.dnetlib.openaire.community.repository.DbSubCommunityRepository;
+import eu.dnetlib.openaire.community.repository.DbSupportOrgRepository;
+import eu.dnetlib.openaire.community.utils.CommunityMappingUtils;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.exceptions.ResourceNotFoundException;
+import eu.dnetlib.openaire.exporter.model.community.CommunityContentprovider;
+import eu.dnetlib.openaire.exporter.model.community.CommunityDetails;
+import eu.dnetlib.openaire.exporter.model.community.CommunityOrganization;
+import eu.dnetlib.openaire.exporter.model.community.CommunityProject;
+import eu.dnetlib.openaire.exporter.model.community.CommunitySummary;
+import eu.dnetlib.openaire.exporter.model.community.CommunityWritableProperties;
+import eu.dnetlib.openaire.exporter.model.community.SubCommunity;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.SelectionCriteria;
+import eu.dnetlib.openaire.exporter.model.context.IISConfigurationEntry;
+
+@Service
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public class CommunityService {
+
+	@Autowired
+	private DbCommunityRepository dbCommunityRepository;
+	@Autowired
+	private DbProjectRepository dbProjectRepository;
+	@Autowired
+	private DbDatasourceRepository dbDatasourceRepository;
+	@Autowired
+	private DbOrganizationRepository dbOrganizationRepository;
+	@Autowired
+	private DbSupportOrgRepository dbSupportOrgRepository;
+	@Autowired
+	private DbSubCommunityRepository dbSubCommunityRepository;
+
+	private static final Log log = LogFactory.getLog(CommunityService.class);
+
+	public List<CommunitySummary> listCommunities() {
+		return dbCommunityRepository.findAll()
+				.stream()
+				.map(CommunityMappingUtils::toCommunitySummary)
+				.collect(Collectors.toList());
+	}
+
+	@Transactional
+	public CommunityDetails newCommunity(final CommunityDetails details) throws CommunityException {
+		if (StringUtils.isBlank(details.getId())) { throw new CommunityException("Empty Id"); }
+		if (dbCommunityRepository.existsById(details.getId())) { throw new CommunityException("Community already exists: " + details.getId()); }
+		details.setCreationDate(LocalDateTime.now());
+		return saveCommunity(details);
+
+	}
+
+	@Transactional
+	public CommunityDetails saveCommunity(final CommunityDetails details) {
+		details.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(CommunityMappingUtils.toCommunity(details));
+		return getCommunity(details.getId());
+	}
+
+	@Transactional
+	public CommunityDetails getCommunity(final String id) {
+		final DbCommunity c = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		return CommunityMappingUtils.toCommunityDetails(c);
+	}
+
+	@Transactional
+	public void setCommunity(final String id, final CommunityWritableProperties details) {
+		final DbCommunity c = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		CommunityMappingUtils.populateCommunity(c, details);
+		c.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(c);
+	}
+
+	@Transactional
+	public Page<CommunityProject> getCommunityProjects(final String id,
+			final String funder,
+			final String filter,
+			final int page,
+			final int size,
+			final String orderBy) throws CommunityException {
+		if (StringUtils.isBlank(id)) { throw new CommunityException("Empty ID"); }
+		try {
+			final Sort sort;
+			if (StringUtils.isBlank(orderBy)) {
+				sort = Sort.by("projectName");
+			} else if ("funder".equalsIgnoreCase(orderBy)) {
+				sort = Sort.by("projectFunder").and(Sort.by("projectName"));
+			} else if ("grantId".equalsIgnoreCase(orderBy)) {
+				sort = Sort.by("projectCode");
+			} else if ("acronym".equalsIgnoreCase(orderBy)) {
+				sort = Sort.by("projectAcronym");
+			} else if ("openaireId".equalsIgnoreCase(orderBy)) {
+				sort = Sort.by("projectId");
+			} else {
+				sort = Sort.by("projectName");
+			}
+
+			final PageRequest pageable = PageRequest.of(page, size, sort);
+			if (StringUtils.isAllBlank(filter, funder)) {
+				return dbProjectRepository.findByCommunity(id, pageable).map(CommunityMappingUtils::toCommunityProject);
+			}
+			final Specification<DbProject> projSpec = prepareProjectSpec(id, funder, filter);
+			return dbProjectRepository.findAll(projSpec, pageable).map(CommunityMappingUtils::toCommunityProject);
+		} catch (final Throwable e) {
+			log.error(e);
+			throw new CommunityException(e);
+		}
+	}
+
+	private Specification<DbProject> prepareProjectSpec(final String community, final String funder, final String other) {
+		return (project, query, cb) -> {
+
+			final List<Predicate> andConds = new ArrayList<>();
+			andConds.add(cb.equal(project.get("community"), community));
+
+			if (StringUtils.isNotBlank(funder)) {
+				andConds.add(cb.equal(project.get("projectFunder"), funder));
+			}
+
+			if (StringUtils.isNotBlank(other)) {
+				final String s = other.toLowerCase().trim();
+
+				final List<Predicate> orConds = new ArrayList<>();
+				orConds.add(cb.equal(cb.lower(project.get("projectId")), s));
+				orConds.add(cb.equal(cb.lower(project.get("projectCode")), s));
+				orConds.add(cb.equal(cb.lower(project.get("projectAcronym")), s));
+				orConds.add(cb.like(cb.lower(project.get("projectName")), "%" + s + "%"));
+				if (StringUtils.isBlank(funder)) {
+					orConds.add(cb.equal(cb.lower(project.get("projectFunder")), s));
+				}
+
+				andConds.add(cb.or(orConds.toArray(new Predicate[orConds.size()])));
+			}
+
+			return cb.and(andConds.toArray(new Predicate[andConds.size()]));
+		};
+
+	}
+
+	@Transactional
+	public CommunityProject addCommunityProject(final String id, final CommunityProject project) {
+		final DbProject p = CommunityMappingUtils.toDbProject(id, project);
+		dbProjectRepository.save(p);
+		return project;
+	}
+
+	@Transactional
+	public void addCommunityProjects(final String id, final CommunityProject... projects) throws CommunityException {
+		try {
+			final List<DbProject> list = Arrays.stream(projects)
+					.map(p -> CommunityMappingUtils.toDbProject(id, p))
+					.collect(Collectors.toList());
+
+			dbProjectRepository.saveAll(list);
+		} catch (final Throwable e) {
+			log.error(e);
+			throw new CommunityException(e);
+		}
+	}
+
+	@Transactional
+	public void removeCommunityProjects(final String id, final String... ids) {
+		final List<DbProjectPK> list = Arrays.stream(ids)
+				.map(projectId -> new DbProjectPK(id, projectId))
+				.collect(Collectors.toList());
+		dbProjectRepository.deleteAllById(list);
+	}
+
+	public List<CommunityContentprovider> getCommunityDatasources(final String id) {
+		return dbDatasourceRepository.findByCommunity(id)
+				.stream()
+				.map(CommunityMappingUtils::toCommunityContentprovider)
+				.collect(Collectors.toList());
+	}
+
+	public List<CommunityContentprovider> getCommunityDatasourcesWithDeposit(final String id, final boolean deposit) {
+		return dbDatasourceRepository.findByCommunityAndDeposit(id, deposit)
+				.stream()
+				.map(CommunityMappingUtils::toCommunityContentprovider)
+				.collect(Collectors.toList());
+	}
+
+	@Transactional
+	public CommunityContentprovider updateCommunityDatasourcesDeposit(final String id, final String dsId, final Boolean deposit, final String message) {
+		return dbDatasourceRepository.findById(new DbDatasourcePK(id, dsId))
+				.map(ds -> {
+					ds.setDeposit(deposit != null ? deposit : false);
+					ds.setMessage(message);
+					return ds;
+				})
+				.map(CommunityMappingUtils::toCommunityContentprovider)
+				.orElseThrow(() -> new ResourceNotFoundException("Community and/or Datasource not found"));
+	}
+
+	@Transactional
+	public void addCommunityDatasources(final String id, final CommunityContentprovider... contentproviders) {
+		final List<DbDatasource> list = Arrays.stream(contentproviders)
+				.map(cp -> CommunityMappingUtils.toDbDatasource(id, cp))
+				.collect(Collectors.toList());
+
+		dbDatasourceRepository.saveAll(list);
+	}
+
+	@Transactional
+	public void removeCommunityDatasources(final String id, final String... ids) {
+		final List<DbDatasourcePK> list = Arrays.stream(ids)
+				.map(dsId -> new DbDatasourcePK(id, dsId))
+				.collect(Collectors.toList());
+		dbDatasourceRepository.deleteAllById(list);
+	}
+
+	@Transactional
+	public void removeCommunityOrganizations(final String id, final String... orgNames) {
+		final List<DbSupportOrgPK> list = Arrays.stream(orgNames)
+				.map(name -> new DbSupportOrgPK(id, name))
+				.collect(Collectors.toList());
+		dbSupportOrgRepository.deleteAllById(list);
+	}
+
+	@Transactional
+	public List<CommunityOrganization> getCommunityOrganizations(final String id) {
+		return dbSupportOrgRepository.findByCommunity(id)
+				.stream()
+				.map(CommunityMappingUtils::toCommunityOrganization)
+				.collect(Collectors.toList());
+	}
+
+	@Transactional
+	public void addCommunityOrganizations(final String id, final CommunityOrganization... orgs) {
+		final List<DbSupportOrg> list = Arrays.stream(orgs)
+				.map(o -> CommunityMappingUtils.toDbSupportOrg(id, o))
+				.collect(Collectors.toList());
+
+		dbSupportOrgRepository.saveAll(list);
+	}
+
+	@Transactional
+	public void removeSubCommunities(final String id, final String... subCommunityIds) {
+		dbSubCommunityRepository.deleteAllById(Arrays.asList(subCommunityIds));
+	}
+
+	@Transactional
+	public List<SubCommunity> getSubCommunities(final String id) {
+		return dbSubCommunityRepository.findByCommunity(id)
+				.stream()
+				.map(CommunityMappingUtils::toSubCommunity)
+				.collect(Collectors.toList());
+	}
+
+	@Transactional
+	public void addSubCommunities(final String id, final SubCommunity... subs) {
+		final List<DbSubCommunity> list = Arrays.stream(subs)
+				.map(s -> CommunityMappingUtils.toDbSubCommunity(id, s))
+				.collect(Collectors.toList());
+
+		dbSubCommunityRepository.saveAll(list);
+	}
+
+	@Transactional
+	public CommunityDetails addCommunitySubjects(final String id, final String... subjects) {
+		return modifyElementToArrayField(id, DbCommunity::getSubjects, DbCommunity::setSubjects, false, subjects);
+	}
+
+	public CommunityDetails removeCommunitySubjects(final String id, final String... subjects) {
+		return modifyElementToArrayField(id, DbCommunity::getSubjects, DbCommunity::setSubjects, true, subjects);
+	}
+
+	public CommunityDetails addCommunityFOS(final String id, final String... foss) {
+		return modifyElementToArrayField(id, DbCommunity::getFos, DbCommunity::setFos, false, foss);
+	}
+
+	public CommunityDetails removeCommunityFOS(final String id, final String... foss) {
+		return modifyElementToArrayField(id, DbCommunity::getFos, DbCommunity::setFos, true, foss);
+	}
+
+	public CommunityDetails addCommunitySDG(final String id, final String... sdgs) {
+		return modifyElementToArrayField(id, DbCommunity::getSdg, DbCommunity::setSdg, false, sdgs);
+	}
+
+	public CommunityDetails removeCommunitySDG(final String id, final String... sdgs) {
+		return modifyElementToArrayField(id, DbCommunity::getSdg, DbCommunity::setSdg, true, sdgs);
+	}
+
+	@Transactional
+	public CommunityDetails addCommunityAdvancedConstraint(final String id, final SelectionCriteria advancedCosntraint) {
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		dbEntry.setAdvancedConstraints(advancedCosntraint);
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(dbEntry);
+		return getCommunity(id);
+	}
+
+	@Transactional
+	public CommunityDetails removeCommunityAdvancedConstraint(final String id) {
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		dbEntry.setAdvancedConstraints(null);
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(dbEntry);
+		return getCommunity(id);
+	}
+
+	@Transactional
+	public CommunityDetails addCommunityRemoveConstraint(final String id, final SelectionCriteria removeConstraint) {
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		dbEntry.setRemoveConstraints(removeConstraint);
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(dbEntry);
+		return getCommunity(id);
+	}
+
+	@Transactional
+	public CommunityDetails removeCommunityRemoveConstraint(final String id) {
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		dbEntry.setRemoveConstraints(null);
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(dbEntry);
+		return getCommunity(id);
+	}
+
+	public CommunityDetails removeCommunityZenodoCommunity(final String id, final String zenodoCommunity, final boolean isMain) {
+		if (isMain) { return updateElementToSimpleField(id, DbCommunity::setMainZenodoCommunity, null); }
+		return modifyElementToArrayField(id, DbCommunity::getOtherZenodoCommunities, DbCommunity::setOtherZenodoCommunities, true, zenodoCommunity);
+	}
+
+	public CommunityDetails addCommunityZenodoCommunity(final String id, final String zenodoCommunity, final boolean isMain) {
+		if (isMain) { return updateElementToSimpleField(id, DbCommunity::setMainZenodoCommunity, zenodoCommunity); }
+		return modifyElementToArrayField(id, DbCommunity::getOtherZenodoCommunities, DbCommunity::setOtherZenodoCommunities, false, zenodoCommunity);
+	}
+
+	@Transactional
+	private CommunityDetails updateElementToSimpleField(final String id,
+			final BiConsumer<DbCommunity, String> setter,
+			final String value) {
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+		setter.accept(dbEntry, value);
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+		dbCommunityRepository.save(dbEntry);
+		return getCommunity(id);
+	}
+
+	@Transactional
+	private CommunityDetails modifyElementToArrayField(final String id,
+			final Function<DbCommunity, String[]> getter,
+			final BiConsumer<DbCommunity, String[]> setter,
+			final boolean remove,
+			final String... values) {
+
+		final DbCommunity dbEntry = dbCommunityRepository.findById(id).orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id));
+
+		final Set<String> tmpList = new LinkedHashSet<>();
+		final String[] oldValues = getter.apply(dbEntry);
+		if (oldValues != null) {
+			Collections.addAll(tmpList, oldValues);
+		}
+		if (remove) {
+			tmpList.removeAll(Arrays.asList(values));
+		} else {
+			tmpList.addAll(Arrays.asList(values));
+		}
+
+		setter.accept(dbEntry, tmpList.toArray(new String[tmpList.size()]));
+
+		dbEntry.setLastUpdateDate(LocalDateTime.now());
+
+		dbCommunityRepository.save(dbEntry);
+
+		return getCommunity(id);
+	}
+
+	@Transactional
+	public List<String> getOpenAIRECommunitiesByZenodoId(final String zenodoId) {
+		return dbCommunityRepository.findByZenodoId(zenodoId);
+	}
+
+	@Transactional
+	public Map<String, Set<String>> getPropagationOrganizationCommunityMap() {
+		return dbOrganizationRepository.findAll()
+				.stream()
+				.collect(Collectors.groupingBy(DbOrganization::getOrgId, Collectors.mapping(DbOrganization::getCommunity, Collectors.toSet())));
+	}
+
+	@Transactional
+	public Set<String> getPropagationOrganizationsForCommunity(final String communityId) {
+		return dbOrganizationRepository.findByCommunity(communityId)
+				.stream()
+				.map(DbOrganization::getOrgId)
+				.collect(Collectors.toSet());
+	}
+
+	@Transactional
+	public Set<String> addPropagationOrganizationForCommunity(final String communityId, final String... organizationIds) {
+		for (final String orgId : organizationIds) {
+			final DbOrganization o = new DbOrganization(communityId.trim(), orgId.trim());
+			dbOrganizationRepository.save(o);
+		}
+		return getPropagationOrganizationsForCommunity(communityId);
+	}
+
+	@Transactional
+	public Set<String> removePropagationOrganizationForCommunity(final String communityId, final String... organizationIds) {
+		for (final String orgId : organizationIds) {
+			final DbOrganization o = new DbOrganization(communityId.trim(), orgId.trim());
+			dbOrganizationRepository.delete(o);
+		}
+		return getPropagationOrganizationsForCommunity(communityId);
+	}
+
+	@Transactional
+	public void deleteCommunity(final String id, final boolean recursive) {
+		if (recursive) {
+			dbProjectRepository.deleteByCommunity(id);
+			dbDatasourceRepository.deleteByCommunity(id);
+			dbOrganizationRepository.deleteByCommunity(id);
+			dbSupportOrgRepository.deleteByCommunity(id);
+			dbSubCommunityRepository.deleteByCommunity(id);
+		}
+		dbCommunityRepository.deleteById(id);
+	}
+
+	@Transactional
+	public List<IISConfigurationEntry> getIISConfiguration(final String id) {
+		final List<IISConfigurationEntry> res = new ArrayList<>();
+
+		res.add(dbCommunityRepository.findById(id)
+				.map(CommunityMappingUtils::asIISConfigurationEntry)
+				.orElseThrow(() -> new ResourceNotFoundException("Community not found: " + id)));
+
+		for (final DbSubCommunity subc : dbSubCommunityRepository.findByCommunity(id)) {
+			res.add(CommunityMappingUtils.asIISConfigurationEntry(subc));
+		}
+
+		return res;
+	}
+
+	@Transactional
+	public List<String> getCommunityFunders(final String id) {
+		return dbProjectRepository.findFundersByCommunity(id);
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/CommunityWritableProperties.java

@@ -1,100 +0,0 @@
-package eu.dnetlib.openaire.community;
-
-import java.util.List;
-
-import com.fasterxml.jackson.annotation.JsonAutoDetect;
-
-import io.swagger.v3.oas.annotations.media.Schema;
-
-@JsonAutoDetect
-public class CommunityWritableProperties {
-
-	@Schema(description = "community name")
-	private String name;
-
-	@Schema(description = "community short name")
-	private String shortName;
-
-	@Schema(description = "community description")
-	private String description;
-
-	@Schema(description = "http url for the community logo")
-	private String logoUrl;
-
-	@Schema(description = "list of subjects (keywords) that characterise this community")
-	private List<String> subjects;
-
-	@Schema(description = "status of the community, drives its visibility")
-	private CommunityStatus status;
-
-	@Schema(description = "id of the main Zenodo community")
-	private String mainZenodoCommunity;
-
-	public static CommunityWritableProperties fromDetails(final CommunityDetails details) {
-		final CommunityWritableProperties p = new CommunityWritableProperties();
-		p.setName(details.getName());
-		p.setShortName(details.getShortName());
-		p.setDescription(details.getDescription());
-		p.setLogoUrl(details.getLogoUrl());
-		p.setSubjects(details.getSubjects());
-		p.setStatus(details.getStatus());
-		p.setMainZenodoCommunity(details.getZenodoCommunity());
-		return p;
-	}
-
-	public String getName() {
-		return name;
-	}
-
-	public void setName(final String name) {
-		this.name = name;
-	}
-
-	public String getShortName() {
-		return shortName;
-	}
-
-	public void setShortName(final String shortName) {
-		this.shortName = shortName;
-	}
-
-	public String getDescription() {
-		return description;
-	}
-
-	public void setDescription(final String description) {
-		this.description = description;
-	}
-
-	public String getLogoUrl() {
-		return logoUrl;
-	}
-
-	public void setLogoUrl(final String logoUrl) {
-		this.logoUrl = logoUrl;
-	}
-
-	public List<String> getSubjects() {
-		return subjects;
-	}
-
-	public void setSubjects(final List<String> subjects) {
-		this.subjects = subjects;
-	}
-
-	public CommunityStatus getStatus() {
-		return status;
-	}
-
-	public void setStatus(final CommunityStatus status) {
-		this.status = status;
-	}
-
-	public String getMainZenodoCommunity() {
-		return mainZenodoCommunity;
-	}
-
-	public void setMainZenodoCommunity(final String mainZenodoCommunity) {
-		this.mainZenodoCommunity = mainZenodoCommunity;
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/importer/CommunityImporterController.java

@@ -0,0 +1,87 @@
+package eu.dnetlib.openaire.community.importer;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.web.bind.annotation.CrossOrigin;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.RestController;
+
+import com.google.common.collect.Sets;
+
+import eu.dnetlib.common.controller.AbstractDnetController;
+import eu.dnetlib.openaire.common.ISClient;
+import eu.dnetlib.openaire.community.model.DbOrganization;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.model.context.Context;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+@RestController
+@CrossOrigin(origins = {
+	"*"
+})
+@ConditionalOnProperty(value = "openaire.exporter.enable.community.import", havingValue = "true")
+@Tag(name = "OpenAIRE Communities: Migration API", description = "OpenAIRE Communities: Migration API")
+public class CommunityImporterController extends AbstractDnetController {
+
+	// public final static Set<String> communityBlackList = Sets.newHashSet("fet-fp7", "fet-h2020");
+	public final static Set<String> communityBlackList = Sets.newHashSet();
+
+	@Autowired
+	private CommunityImporterService importer;
+
+	@Autowired
+	private ISClient isClient;
+
+	private static final Log log = LogFactory.getLog(CommunityImporterController.class);
+
+	@GetMapping("/community_importer/communities")
+	public List<String> importProfiles() throws CommunityException {
+		try {
+			final Map<String, Context> contextMap = getContextMap();
+
+			final List<String> list = contextMap.keySet()
+				.stream()
+				.filter(id -> !communityBlackList.contains(id))
+				.collect(Collectors.toList());
+
+			list.forEach(id -> {
+				importer.importCommunity(contextMap.get(id));
+			});
+
+			return list;
+		} catch (final Throwable e) {
+			log.error("Error importing communities", e);
+			throw new CommunityException(e.getMessage());
+		}
+	}
+
+	@GetMapping("/community_importer/propagationOrgs")
+	public List<DbOrganization> importPropagationOrgs(@RequestParam final String profileId,
+		@RequestParam(required = false, defaultValue = "false") final boolean simulation) throws Exception {
+		try {
+			final String xml = isClient.getProfile(profileId);
+			return importer.importPropagationOrganizationsFromProfile(xml, simulation);
+		} catch (final Throwable e) {
+			log.error("Error importing communities", e);
+			throw new CommunityException(e.getMessage());
+		}
+	}
+
+	private Map<String, Context> getContextMap() throws CommunityException {
+		try {
+			return isClient.getCommunityContextMap();
+		} catch (final IOException e) {
+			throw new CommunityException(e);
+		}
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/importer/CommunityImporterService.java

@@ -0,0 +1,411 @@
+package eu.dnetlib.openaire.community.importer;
+
+import java.time.LocalDate;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import javax.transaction.Transactional;
+
+import org.apache.commons.lang3.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.dom4j.DocumentHelper;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.stereotype.Service;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.collect.Lists;
+
+import eu.dnetlib.miscutils.functional.hash.Hashing;
+import eu.dnetlib.openaire.community.CommunityService;
+import eu.dnetlib.openaire.community.model.DbOrganization;
+import eu.dnetlib.openaire.community.repository.DbOrganizationRepository;
+import eu.dnetlib.openaire.community.utils.CommunityMappingUtils;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.model.community.CommunityClaimType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityContentprovider;
+import eu.dnetlib.openaire.exporter.model.community.CommunityDetails;
+import eu.dnetlib.openaire.exporter.model.community.CommunityMembershipType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityOrganization;
+import eu.dnetlib.openaire.exporter.model.community.CommunityPlanType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityProject;
+import eu.dnetlib.openaire.exporter.model.community.CommunityStatus;
+import eu.dnetlib.openaire.exporter.model.community.CommunityType;
+import eu.dnetlib.openaire.exporter.model.community.SubCommunity;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.SelectionCriteria;
+import eu.dnetlib.openaire.exporter.model.context.Category;
+import eu.dnetlib.openaire.exporter.model.context.Concept;
+import eu.dnetlib.openaire.exporter.model.context.Context;
+import eu.dnetlib.openaire.exporter.model.context.Param;
+
+@Service
+@ConditionalOnProperty(value = "openaire.exporter.enable.community.import", havingValue = "true")
+public class CommunityImporterService {
+
+	// common
+	public final static String OPENAIRE_ID = "openaireId";
+	public final static String PIPE_SEPARATOR = "||";
+	public final static String ID_SEPARATOR = "::";
+	public final static String CSV_DELIMITER = ",";
+	public final static String CLABEL = "label";
+
+	// id suffixes
+	public final static String PROJECTS_ID_SUFFIX = ID_SEPARATOR + "projects";
+	public final static String CONTENTPROVIDERS_ID_SUFFIX = ID_SEPARATOR + "contentproviders";
+	public final static String ZENODOCOMMUNITY_ID_SUFFIX = ID_SEPARATOR + "zenodocommunities";
+	public final static String ORGANIZATION_ID_SUFFIX = ID_SEPARATOR + "organizations";
+
+	// community summary
+	public final static String CSUMMARY_DESCRIPTION = "description";
+	public final static String CSUMMARY_LOGOURL = "logourl";
+	public final static String CSUMMARY_STATUS = "status";
+	public final static String CSUMMARY_NAME = "name";
+	public final static String CSUMMARY_MANAGER = "manager";
+	public final static String CSUMMARY_ZENODOC = "zenodoCommunity";
+
+	// community profile
+	public final static String CPROFILE_SUBJECT = "subject";
+	public final static String CPROFILE_CREATIONDATE = "creationdate";
+	public final static String CPROFILE_FOS = "fos";
+	public final static String CPROFILE_SDG = "sdg";
+	public final static String CPROFILE_ADVANCED_CONSTRAINT = "advancedConstraints";
+	public final static String CPROFILE_REMOVE_CONSTRAINT = "removeConstraints";
+	public final static String CPROFILE_SUGGESTED_ACKNOWLEDGEMENT = "suggestedAcknowledgement";
+
+	// community project
+	public final static String CPROJECT_FUNDER = "funder";
+	public final static String CPROJECT_NUMBER = "CD_PROJECT_NUMBER";
+	public final static String CPROJECT_FULLNAME = "projectfullname";
+	public final static String CPROJECT_ACRONYM = "acronym";
+
+	// community content provider
+	public final static String CCONTENTPROVIDER_NAME = "name";
+	public final static String CCONTENTPROVIDER_OFFICIALNAME = "officialname";
+	public final static String CCONTENTPROVIDER_ENABLED = "enabled";
+	public final static String CCONTENTPROVIDERENABLED_DEFAULT = "true";
+	public final static String CCONTENTPROVIDER_SELCRITERIA = "selcriteria";
+
+	// community zenodo community
+	public final static String CZENODOCOMMUNITY_ID = "zenodoid";
+
+	// community organization
+	public final static String CORGANIZATION_NAME = "name";
+	public final static String CORGANIZATION_LOGOURL = "logourl";
+	public final static String CORGANIZATION_WEBSITEURL = "websiteurl";
+
+	@Autowired
+	private DbOrganizationRepository dbOrganizationRepository;
+
+	@Autowired
+	private CommunityService service;
+
+	@Autowired
+	private JdbcTemplate jdbcTemplate;
+
+	private static final Log log = LogFactory.getLog(CommunityImporterService.class);
+
+	public List<DbOrganization> importPropagationOrganizationsFromProfile(final String xml, final boolean simulation) throws Exception {
+		final String json = DocumentHelper.parseText(xml)
+				.selectSingleNode("//NODE[@name='setPropagationOrganizationCommunityMap']//PARAM[@name='parameterValue']")
+				.getText();
+
+		final List<DbOrganization> list = new ObjectMapper()
+				.readValue(json, new TypeReference<Map<String, List<String>>>() {})
+				.entrySet()
+				.stream()
+				.flatMap(e -> e.getValue()
+						.stream()
+						.map(community -> {
+							if (e.getKey().contains("|")) { return new DbOrganization(community, StringUtils.substringAfter(e.getKey(), "|")); }
+							return new DbOrganization(community, e.getKey());
+						}))
+				.collect(Collectors.toList());
+
+		if (!simulation) {
+			list.forEach(o -> {
+				try {
+					dbOrganizationRepository.save(o);
+				} catch (final Throwable e) {
+					log.error("ERROR saving org: " + o);
+				}
+			});
+		}
+
+		return list;
+	}
+
+	@Transactional
+	public void importCommunity(final Context context) {
+		try {
+
+			final CommunityDetails community = asCommunityDetails(context);
+
+			final List<CommunityContentprovider> datasources =
+					getCommunityInfo(context, CONTENTPROVIDERS_ID_SUFFIX, c -> asCommunityDataprovider(context.getId(), c))
+							.stream()
+							.map(o -> {
+								if (o.getOpenaireId() == null) {
+									log.warn("Openaire ID is missing, organization: " + o.getOfficialname());
+								} else if (o.getOpenaireId().contains("|")) {
+									o.setOpenaireId(StringUtils.substringAfter(o.getOpenaireId(), "|"));
+								}
+								return o;
+							})
+							.filter(o -> o.getOpenaireId() != null)
+							.collect(Collectors.toList());
+
+			final List<CommunityProject> projects =
+					getCommunityInfo(context, PROJECTS_ID_SUFFIX, c -> asCommunityProject(context.getId(), c))
+							.stream()
+							.map(p -> {
+								if (p.getOpenaireId() == null) {
+									if ("EC".equalsIgnoreCase(p.getFunder())) {
+										final String ns = findNamespaceForECProject(p.getGrantId());
+										if (ns != null) {
+											p.setOpenaireId(ns + "::" + Hashing.md5(p.getGrantId()));
+										} else {
+											log.warn("EC project not in the db: " + p.getGrantId());
+										}
+									} else if ("NSF".equalsIgnoreCase(p.getFunder())) {
+										p.setOpenaireId("nsf_________::" + Hashing.md5(p.getGrantId()));
+									} else if ("NIH".equalsIgnoreCase(p.getFunder())) {
+										p.setOpenaireId("nih_________::" + Hashing.md5(p.getGrantId()));
+									} else {
+										log.warn("Openaire ID is missing, funder: " + p.getFunder());
+									}
+								} else if (p.getOpenaireId().contains("|")) {
+									p.setOpenaireId(StringUtils.substringAfter(p.getOpenaireId(), "|"));
+								}
+								return p;
+							})
+							.filter(p -> p.getOpenaireId() != null)
+							.collect(Collectors.toList());
+
+			final List<CommunityOrganization> orgs =
+					getCommunityInfo(context, ORGANIZATION_ID_SUFFIX, c -> asCommunityOrganization(context.getId(), c));
+
+			final List<String> otherZenodoCommunities =
+					getCommunityInfo(context, ZENODOCOMMUNITY_ID_SUFFIX, CommunityImporterService::asZenodoCommunity);
+
+			community.setOtherZenodoCommunities(otherZenodoCommunities);
+
+			final List<SubCommunity> subs = context.getCategories()
+					.entrySet()
+					.stream()
+					.filter(e -> !(context.getId() + CONTENTPROVIDERS_ID_SUFFIX).equals(e.getKey()))
+					.filter(e -> !(context.getId() + PROJECTS_ID_SUFFIX).equals(e.getKey()))
+					.filter(e -> !(context.getId() + ORGANIZATION_ID_SUFFIX).equals(e.getKey()))
+					.filter(e -> !(context.getId() + ZENODOCOMMUNITY_ID_SUFFIX).equals(e.getKey()))
+					.map(Entry::getValue)
+					.map(cat -> asSubCommunities(context.getId(), null, cat.getLabel(), cat.getConcepts()))
+					.flatMap(List::stream)
+					.collect(Collectors.toList());
+
+			service.saveCommunity(community);
+			service.addCommunityProjects(context.getId(), projects.toArray(new CommunityProject[projects.size()]));
+			service.addCommunityDatasources(context.getId(), datasources.toArray(new CommunityContentprovider[datasources.size()]));
+			service.addCommunityOrganizations(context.getId(), orgs.toArray(new CommunityOrganization[orgs.size()]));
+			service.addSubCommunities(context.getId(), subs.toArray(new SubCommunity[subs.size()]));
+		} catch (
+
+		final Exception e) {
+			throw new RuntimeException("Error importing community: " + context.getId(), e);
+		}
+	}
+
+	private <R> List<R> getCommunityInfo(final Context context, final String idSuffix, final Function<Concept, R> mapping)
+			throws CommunityException {
+		if (context != null) {
+			final Map<String, Category> categories = context.getCategories();
+			final Category category = categories.get(context.getId() + idSuffix);
+			if (category != null) { return category.getConcepts()
+					.stream()
+					.map(mapping)
+					.collect(Collectors.toList()); }
+		}
+		return Lists.newArrayList();
+	}
+
+	private static CommunityDetails asCommunityDetails(final Context c) {
+
+		final CommunityDetails details = new CommunityDetails();
+
+		details.setId(c.getId());
+		details.setShortName(c.getLabel());
+		details.setDisplayShortName(c.getLabel());
+		details.setLastUpdateDate(CommunityMappingUtils.asLocalDateTime(c.getLastUpdateDate()));
+		details.setCreationDate(CommunityMappingUtils.asLocalDateTime(c.getCreationDate()));
+		details.setQueryId(c.getId() + PIPE_SEPARATOR + c.getLabel());
+		details.setType(CommunityType.valueOf(c.getType()));
+		details.setMembership(CommunityMembershipType.open);
+		details.setClaim(CommunityClaimType.all);
+		details.setDescription(asCsv(CSUMMARY_DESCRIPTION, c.getParams()));
+		details.setLogoUrl(asCsv(CSUMMARY_LOGOURL, c.getParams()));
+
+		final String status = firstValue(CSUMMARY_STATUS, c.getParams());
+		if (StringUtils.isNotBlank(status)) {
+			details.setStatus(CommunityStatus.valueOf(status));
+		} else {
+			details.setStatus(CommunityStatus.hidden);
+		}
+
+		details.setName(StringUtils.firstNonBlank(asCsv(CSUMMARY_NAME, c.getParams()), c.getLabel()));
+		details.setDisplayName(StringUtils.firstNonBlank(asCsv(CSUMMARY_NAME, c.getParams()), c.getLabel()));
+
+		details.setZenodoCommunity(asCsv(CSUMMARY_ZENODOC, c.getParams()));
+		details.setSubjects(splitValues(asValues(CPROFILE_SUBJECT, c.getParams()), CSV_DELIMITER));
+		details.setFos(splitValues(asValues(CPROFILE_FOS, c.getParams()), CSV_DELIMITER));
+		details.setSdg(splitValues(asValues(CPROFILE_SDG, c.getParams()), CSV_DELIMITER));
+		// In the map the string is the serialization of the json representing the selection criteria so it is a valid json
+		details.setAdvancedConstraints(SelectionCriteria.fromJson(asCsv(CPROFILE_ADVANCED_CONSTRAINT, c.getParams())));
+		// In the map the string is the serialization of the json representing the selection criteria so it is a valid json
+		details.setRemoveConstraints(SelectionCriteria.fromJson(asCsv(CPROFILE_REMOVE_CONSTRAINT, c.getParams())));
+		details.setSuggestedAcknowledgements(splitValues(asValues(CPROFILE_SUGGESTED_ACKNOWLEDGEMENT, c.getParams()), CSV_DELIMITER));
+		details.setPlan(CommunityPlanType.Default);
+		try {
+			details.setCreationDate(CommunityMappingUtils.asLocalDateTime(asCsv(CPROFILE_CREATIONDATE, c.getParams())));
+		} catch (final Exception e) {
+			log.debug("Exception on date format: " + e.getMessage());
+		}
+
+		return details;
+	}
+
+	private static CommunityProject asCommunityProject(final String communityId, final Concept c) {
+		final List<Param> p = c.getParams();
+		final CommunityProject project = new CommunityProject();
+		project.setCommunityId(communityId);
+		project.setOpenaireId(firstValue(OPENAIRE_ID, p));
+		project.setFunder(firstValue(CPROJECT_FUNDER, p));
+		project.setGrantId(firstValue(CPROJECT_NUMBER, p));
+		project.setName(firstValue(CPROJECT_FULLNAME, p));
+		project.setAcronym(firstValue(CPROJECT_ACRONYM, p));
+		project.setAvailableSince(LocalDate.of(2017, 2, 25)); // Birillo Birth Date
+		return project;
+	}
+
+	private static CommunityContentprovider asCommunityDataprovider(final String communityId, final Concept c) {
+		final List<Param> p = c.getParams();
+		final CommunityContentprovider d = new CommunityContentprovider();
+		d.setCommunityId(communityId);
+		d.setOpenaireId(firstValue(OPENAIRE_ID, p));
+		d.setName(firstValue(CCONTENTPROVIDER_NAME, p));
+		d.setOfficialname(firstValue(CCONTENTPROVIDER_OFFICIALNAME, p));
+		d.setEnabled(BooleanUtils.toBoolean(firstValue(CCONTENTPROVIDER_ENABLED, p)));
+		d.setSelectioncriteria(SelectionCriteria.fromJson(firstValue(CCONTENTPROVIDER_SELCRITERIA, p)));
+		d.setDeposit(false);
+		d.setMessage(null);
+		return d;
+	}
+
+	private static CommunityOrganization asCommunityOrganization(final String id, final Concept c) {
+		final List<Param> p = c.getParams();
+		final CommunityOrganization o = new CommunityOrganization();
+		o.setCommunityId(id);
+		o.setName(firstValue(CORGANIZATION_NAME, p));
+		o.setLogo_url(getDecodedUrl(firstValue(CORGANIZATION_LOGOURL, p)));
+		o.setWebsite_url(getDecodedUrl(firstValue(CORGANIZATION_WEBSITEURL, p)));
+		return o;
+	}
+
+	private static String asZenodoCommunity(final Concept c) {
+		return firstValue(CZENODOCOMMUNITY_ID, c.getParams());
+	}
+
+	private static List<SubCommunity> asSubCommunities(final String communityId, final String parent, final String category, final List<Concept> concepts) {
+		final List<SubCommunity> list = new ArrayList<>();
+		for (final Concept c : concepts) {
+			final SubCommunity sc = new SubCommunity();
+			sc.setSubCommunityId(c.getId());
+			sc.setCommunityId(communityId);
+			sc.setParent(parent);
+			sc.setCategory(category);
+			sc.setLabel(c.getLabel());
+			sc.setParams(c.getParams());
+			sc.setClaim(c.isClaim());
+			sc.setBrowsable(false);
+			list.add(sc);
+			list.addAll(asSubCommunities(communityId, c.getId(), category, c.getConcepts()));
+		}
+		return list;
+	}
+
+	private String findNamespaceForECProject(final String code) {
+		final List<String> list =
+				jdbcTemplate.queryForList("SELECT substr(id, 1, 12) from projects where code = ? and id like 'corda%'", String.class, code);
+		return list.isEmpty() ? null : list.get(0);
+	}
+
+	private static String getDecodedUrl(final String encoded_url) {
+		if (encoded_url == null || encoded_url.startsWith("http")) { return encoded_url; }
+		try {
+			return new String(Base64.getDecoder().decode(encoded_url));
+		} catch (final Exception e) {
+			log.warn("Invalid base64: " + encoded_url);
+			return encoded_url;
+		}
+	}
+
+	private static List<String> splitValues(final Stream<String> stream, final String separator) {
+		return stream.map(s -> s.split(separator))
+				.map(Arrays::asList)
+				.flatMap(List::stream)
+				.filter(StringUtils::isNotBlank)
+				.map(StringUtils::trim)
+				.collect(Collectors.toList());
+	}
+
+	private static String firstValue(final String name, final List<Param> params) {
+		return asValues(name, params).findFirst().orElse(null);
+	}
+
+	private static String asCsv(final String name, final List<Param> params) {
+		return asValues(name, params).collect(Collectors.joining(CSV_DELIMITER));
+	}
+
+	private static Stream<String> asValues(final String name, final List<Param> params) {
+		return params == null ? Stream.empty()
+				: params.stream()
+						.filter(p -> p != null)
+						.filter(p -> StringUtils.isNotBlank(p.getName()))
+						.filter(p -> p.getName().trim().equals(name.trim()))
+						.map(Param::getValue)
+						.map(StringUtils::trim)
+						.distinct();
+	}
+
+	protected DbOrganizationRepository getDbOrganizationRepository() {
+		return dbOrganizationRepository;
+	}
+
+	protected void setDbOrganizationRepository(final DbOrganizationRepository dbOrganizationRepository) {
+		this.dbOrganizationRepository = dbOrganizationRepository;
+	}
+
+	protected CommunityService getService() {
+		return service;
+	}
+
+	protected void setService(final CommunityService service) {
+		this.service = service;
+	}
+
+	protected JdbcTemplate getJdbcTemplate() {
+		return jdbcTemplate;
+	}
+
+	protected void setJdbcTemplate(final JdbcTemplate jdbcTemplate) {
+		this.jdbcTemplate = jdbcTemplate;
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbCommunity.java

@@ -0,0 +1,304 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.time.LocalDateTime;
+
+import javax.persistence.Column;
+import javax.persistence.Convert;
+import javax.persistence.Entity;
+import javax.persistence.EnumType;
+import javax.persistence.Enumerated;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.Id;
+import javax.persistence.Table;
+
+import org.hibernate.annotations.Type;
+import org.hibernate.annotations.TypeDef;
+import org.hibernate.annotations.TypeDefs;
+import org.springframework.data.annotation.CreatedDate;
+import org.springframework.data.annotation.LastModifiedDate;
+
+import com.vladmihalcea.hibernate.type.array.StringArrayType;
+import com.vladmihalcea.hibernate.type.json.JsonBinaryType;
+import com.vladmihalcea.hibernate.type.json.JsonStringType;
+
+import eu.dnetlib.openaire.community.utils.CommunityClaimTypeConverter;
+import eu.dnetlib.openaire.community.utils.CommunityMembershipTypeConverter;
+import eu.dnetlib.openaire.exporter.model.community.CommunityClaimType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityMembershipType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityPlanType;
+import eu.dnetlib.openaire.exporter.model.community.CommunityStatus;
+import eu.dnetlib.openaire.exporter.model.community.CommunityType;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.SelectionCriteria;
+
+@Entity
+@Table(name = "communities")
+@TypeDefs({
+		@TypeDef(name = "string-array", typeClass = StringArrayType.class),
+		@TypeDef(name = "json", typeClass = JsonStringType.class),
+		@TypeDef(name = "jsonb", typeClass = JsonBinaryType.class)
+})
+public class DbCommunity implements Serializable {
+
+	private static final long serialVersionUID = 4315597783109726539L;
+
+	@Id
+	@Column(name = "id")
+	private String id;
+
+	@Column(name = "name")
+	private String name;
+
+	@Column(name = "shortname")
+	private String shortName;
+
+	@Column(name = "displayname")
+	private String displayName;
+
+	@Column(name = "displayshortname")
+	private String displayShortName;
+
+	@Column(name = "description")
+	private String description;
+
+	@Column(name = "status")
+	@Enumerated(EnumType.STRING)
+	private CommunityStatus status = CommunityStatus.hidden;
+
+	@Column(name = "membership")
+	@Convert(converter = CommunityMembershipTypeConverter.class)
+	private CommunityMembershipType membership = CommunityMembershipType.byInvitation;
+
+	@Column(name = "type")
+	@Enumerated(EnumType.STRING)
+	private CommunityType type;
+
+	@Column(name = "claim")
+	@Convert(converter = CommunityClaimTypeConverter.class)
+	private CommunityClaimType claim;
+
+	@Type(type = "string-array")
+	@Column(name = "subjects", columnDefinition = "text[]")
+	private String[] subjects;
+
+	@Type(type = "string-array")
+	@Column(name = "fos", columnDefinition = "text[]")
+	private String[] fos;
+
+	@Type(type = "string-array")
+	@Column(name = "sdg", columnDefinition = "text[]")
+	private String[] sdg;
+
+	@Type(type = "jsonb")
+	@Column(name = "adv_constraints")
+	private SelectionCriteria advancedConstraints;
+
+	@Type(type = "jsonb")
+	@Column(name = "remove_constraints")
+	private SelectionCriteria removeConstraints;
+
+	@Column(name = "main_zenodo_community")
+	private String mainZenodoCommunity;
+
+	@Type(type = "string-array")
+	@Column(name = "other_zenodo_communities", columnDefinition = "text[]")
+	private String[] otherZenodoCommunities;
+
+	@CreatedDate
+	@Column(name = "creation_date")
+	private LocalDateTime creationDate;
+
+	@LastModifiedDate
+	@Column(name = "last_update")
+	@GeneratedValue(strategy = GenerationType.IDENTITY)
+	private LocalDateTime lastUpdateDate;
+
+	@Column(name = "logo_url")
+	private String logoUrl;
+
+	@Type(type = "string-array")
+	@Column(name = "suggested_acknowledgements", columnDefinition = "text[]")
+	private String[] suggestedAcknowledgements;
+
+	@Column(name = "plan")
+	@Enumerated(EnumType.STRING)
+	private CommunityPlanType plan;
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(final String id) {
+		this.id = id;
+	}
+
+	public String getName() {
+		return name;
+	}
+
+	public void setName(final String name) {
+		this.name = name;
+	}
+
+	public String getShortName() {
+		return shortName;
+	}
+
+	public void setShortName(final String shortName) {
+		this.shortName = shortName;
+	}
+
+	public String getDisplayName() {
+		return displayName;
+	}
+
+	public void setDisplayName(final String displayName) {
+		this.displayName = displayName;
+	}
+
+	public String getDisplayShortName() {
+		return displayShortName;
+	}
+
+	public void setDisplayShortName(final String displayShortName) {
+		this.displayShortName = displayShortName;
+	}
+
+	public String getDescription() {
+		return description;
+	}
+
+	public void setDescription(final String description) {
+		this.description = description;
+	}
+
+	public CommunityStatus getStatus() {
+		return status;
+	}
+
+	public void setStatus(final CommunityStatus status) {
+		this.status = status;
+	}
+
+	public CommunityMembershipType getMembership() {
+		return membership;
+	}
+
+	public void setMembership(final CommunityMembershipType membership) {
+		this.membership = membership;
+	}
+
+	public CommunityType getType() {
+		return type;
+	}
+
+	public void setType(final CommunityType type) {
+		this.type = type;
+	}
+
+	public CommunityClaimType getClaim() {
+		return claim;
+	}
+
+	public void setClaim(final CommunityClaimType claim) {
+		this.claim = claim;
+	}
+
+	public String[] getSubjects() {
+		return subjects;
+	}
+
+	public void setSubjects(final String[] subjects) {
+		this.subjects = subjects;
+	}
+
+	public String[] getFos() {
+		return fos;
+	}
+
+	public void setFos(final String[] fos) {
+		this.fos = fos;
+	}
+
+	public String[] getSdg() {
+		return sdg;
+	}
+
+	public void setSdg(final String[] sdg) {
+		this.sdg = sdg;
+	}
+
+	public SelectionCriteria getAdvancedConstraints() {
+		return advancedConstraints;
+	}
+
+	public void setAdvancedConstraints(final SelectionCriteria advancedConstraints) {
+		this.advancedConstraints = advancedConstraints;
+	}
+
+	public SelectionCriteria getRemoveConstraints() {
+		return removeConstraints;
+	}
+
+	public void setRemoveConstraints(final SelectionCriteria removeConstraints) {
+		this.removeConstraints = removeConstraints;
+	}
+
+	public String getMainZenodoCommunity() {
+		return mainZenodoCommunity;
+	}
+
+	public void setMainZenodoCommunity(final String mainZenodoCommunity) {
+		this.mainZenodoCommunity = mainZenodoCommunity;
+	}
+
+	public String[] getOtherZenodoCommunities() {
+		return otherZenodoCommunities;
+	}
+
+	public void setOtherZenodoCommunities(final String[] otherZenodoCommunities) {
+		this.otherZenodoCommunities = otherZenodoCommunities;
+	}
+
+	public LocalDateTime getCreationDate() {
+		return creationDate;
+	}
+
+	public void setCreationDate(final LocalDateTime creationDate) {
+		this.creationDate = creationDate;
+	}
+
+	public LocalDateTime getLastUpdateDate() {
+		return lastUpdateDate;
+	}
+
+	public void setLastUpdateDate(final LocalDateTime lastUpdateDate) {
+		this.lastUpdateDate = lastUpdateDate;
+	}
+
+	public String getLogoUrl() {
+		return logoUrl;
+	}
+
+	public void setLogoUrl(final String logoUrl) {
+		this.logoUrl = logoUrl;
+	}
+
+	public String[] getSuggestedAcknowledgements() {
+		return suggestedAcknowledgements;
+	}
+
+	public void setSuggestedAcknowledgements(final String[] suggestedAcknowledgements) {
+		this.suggestedAcknowledgements = suggestedAcknowledgements;
+	}
+
+	public CommunityPlanType getPlan() {
+		return plan;
+	}
+
+	public void setPlan(final CommunityPlanType plan) {
+		this.plan = plan;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbDatasource.java

@@ -0,0 +1,123 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Table;
+
+import org.hibernate.annotations.Type;
+
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.SelectionCriteria;
+
+@Entity
+@Table(name = "community_datasources")
+@IdClass(DbDatasourcePK.class)
+public class DbDatasource implements Serializable {
+
+	private static final long serialVersionUID = -8782576185861694228L;
+
+	@Id
+	@Column(name = "community")
+	private String community;
+
+	@Id
+	@Column(name = "ds_id")
+	private String dsId;
+
+	@Column(name = "ds_name")
+	private String dsName;
+
+	@Column(name = "ds_officialname")
+	private String dsOfficialName;
+
+	@Column(name = "enabled")
+	private Boolean enabled;
+
+	@Type(type = "jsonb")
+	@Column(name = "constraints")
+	private SelectionCriteria constraints;
+
+	@Column(name = "deposit")
+	private Boolean deposit;
+
+	@Column(name = "message")
+	private String message;
+
+	public DbDatasource() {}
+
+	public DbDatasource(final String community, final String dsId, final String dsName, final String dsOfficialName, final SelectionCriteria constraints) {
+		this.community = community;
+		this.dsId = dsId;
+		this.dsName = dsName;
+		this.dsOfficialName = dsOfficialName;
+		this.constraints = constraints;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getDsId() {
+		return dsId;
+	}
+
+	public void setDsId(final String dsId) {
+		this.dsId = dsId;
+	}
+
+	public String getDsName() {
+		return dsName;
+	}
+
+	public void setDsName(final String dsName) {
+		this.dsName = dsName;
+	}
+
+	public String getDsOfficialName() {
+		return dsOfficialName;
+	}
+
+	public void setDsOfficialName(final String dsOfficialName) {
+		this.dsOfficialName = dsOfficialName;
+	}
+
+	public Boolean getEnabled() {
+		return enabled;
+	}
+
+	public void setEnabled(final Boolean enabled) {
+		this.enabled = enabled;
+	}
+
+	public SelectionCriteria getConstraints() {
+		return constraints;
+	}
+
+	public void setConstraints(final SelectionCriteria constraints) {
+		this.constraints = constraints;
+	}
+
+	public Boolean getDeposit() {
+		return deposit;
+	}
+
+	public void setDeposit(final Boolean deposit) {
+		this.deposit = deposit;
+	}
+
+	public String getMessage() {
+		return message;
+	}
+
+	public void setMessage(final String message) {
+		this.message = message;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbDatasourcePK.java

@@ -0,0 +1,55 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class DbDatasourcePK implements Serializable {
+
+	private static final long serialVersionUID = -8073510491611213955L;
+
+	private String community;
+
+	private String dsId;
+
+	public DbDatasourcePK() {}
+
+	public DbDatasourcePK(final String community, final String dsId) {
+		this.community = community;
+		this.dsId = dsId;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getDsId() {
+		return dsId;
+	}
+
+	public void setDsId(final String dsId) {
+		this.dsId = dsId;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(community, dsId);
+	}
+
+	@Override
+	public boolean equals(final Object obj) {
+		if (this == obj) { return true; }
+		if (!(obj instanceof DbDatasourcePK)) { return false; }
+		final DbDatasourcePK other = (DbDatasourcePK) obj;
+		return Objects.equals(community, other.community) && Objects.equals(dsId, other.dsId);
+	}
+
+	@Override
+	public String toString() {
+		return String.format("CommunityDatasourcePK [community=%s, dsId=%s]", community, dsId);
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbOrganization.java

@@ -0,0 +1,54 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Table;
+
+@Entity
+@Table(name = "community_orgs")
+@IdClass(DbOrganizationPK.class)
+public class DbOrganization implements Serializable {
+
+	private static final long serialVersionUID = -602114117980437763L;
+
+	@Id
+	@Column(name = "community")
+	private String community;
+
+	@Id
+	@Column(name = "org_id")
+	private String orgId;
+
+	public DbOrganization() {}
+
+	public DbOrganization(final String community, final String orgId) {
+		this.community = community;
+		this.orgId = orgId;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getOrgId() {
+		return orgId;
+	}
+
+	public void setOrgId(final String orgId) {
+		this.orgId = orgId;
+	}
+
+	@Override
+	public String toString() {
+		return String.format("DbOrganization [community=%s, orgId=%s]", community, orgId);
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbOrganizationPK.java

@@ -0,0 +1,54 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class DbOrganizationPK implements Serializable {
+
+	private static final long serialVersionUID = -6720182815397534837L;
+
+	private String community;
+
+	private String orgId;
+
+	public DbOrganizationPK() {}
+
+	public DbOrganizationPK(final String community, final String orgId) {
+		this.community = community;
+		this.orgId = orgId;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getOrgId() {
+		return orgId;
+	}
+
+	public void setOrgId(final String orgId) {
+		this.orgId = orgId;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(community, orgId);
+	}
+
+	@Override
+	public boolean equals(final Object obj) {
+		if (this == obj) { return true; }
+		if (!(obj instanceof DbDatasourcePK)) { return false; }
+		final DbOrganizationPK other = (DbOrganizationPK) obj;
+		return Objects.equals(community, other.community) && Objects.equals(orgId, other.orgId);
+	}
+
+	@Override
+	public String toString() {
+		return String.format("CommunityOrgPK [community=%s, orgId=%s]", community, orgId);
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbProject.java

@@ -0,0 +1,113 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.time.LocalDate;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Table;
+
+import org.springframework.data.annotation.CreatedDate;
+
+@Entity
+@Table(name = "community_projects")
+@IdClass(DbProjectPK.class)
+public class DbProject implements Serializable {
+
+	private static final long serialVersionUID = 1649065971750517925L;
+
+	@Id
+	@Column(name = "community")
+	private String community;
+
+	@Id
+	@Column(name = "project_id")
+	private String projectId;
+
+	@Column(name = "project_code")
+	private String projectCode;
+
+	@Column(name = "project_name")
+	private String projectName;
+
+	@Column(name = "project_acronym")
+	private String projectAcronym;
+
+	@Column(name = "project_funder")
+	private String projectFunder;
+
+	@CreatedDate
+	@Column(name = "available_since")
+	private LocalDate availableSince;
+
+	public DbProject() {}
+
+	public DbProject(final String community, final String projectId, final String projectCode, final String projectName, final String projectAcronym,
+		final String projectFunder, final LocalDate availableSince) {
+		this.community = community;
+		this.projectId = projectId;
+		this.projectCode = projectCode;
+		this.projectName = projectName;
+		this.projectAcronym = projectAcronym;
+		this.projectFunder = projectFunder;
+		this.availableSince = availableSince;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getProjectId() {
+		return projectId;
+	}
+
+	public void setProjectId(final String projectId) {
+		this.projectId = projectId;
+	}
+
+	public String getProjectCode() {
+		return projectCode;
+	}
+
+	public void setProjectCode(final String projectCode) {
+		this.projectCode = projectCode;
+	}
+
+	public String getProjectName() {
+		return projectName;
+	}
+
+	public void setProjectName(final String projectName) {
+		this.projectName = projectName;
+	}
+
+	public String getProjectAcronym() {
+		return projectAcronym;
+	}
+
+	public void setProjectAcronym(final String projectAcronym) {
+		this.projectAcronym = projectAcronym;
+	}
+
+	public String getProjectFunder() {
+		return projectFunder;
+	}
+
+	public void setProjectFunder(final String projectFunder) {
+		this.projectFunder = projectFunder;
+	}
+
+	public LocalDate getAvailableSince() {
+		return availableSince;
+	}
+
+	public void setAvailableSince(final LocalDate availableSince) {
+		this.availableSince = availableSince;
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbProjectPK.java

@@ -0,0 +1,54 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class DbProjectPK implements Serializable {
+
+	private static final long serialVersionUID = -4236577148534835803L;
+
+	private String community;
+
+	private String projectId;
+
+	public DbProjectPK() {}
+
+	public DbProjectPK(final String community, final String projectId) {
+		this.community = community;
+		this.projectId = projectId;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getProjectId() {
+		return projectId;
+	}
+
+	public void setProjectId(final String projectId) {
+		this.projectId = projectId;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(community, projectId);
+	}
+
+	@Override
+	public boolean equals(final Object obj) {
+		if (this == obj) { return true; }
+		if (!(obj instanceof DbProjectPK)) { return false; }
+		final DbProjectPK other = (DbProjectPK) obj;
+		return Objects.equals(community, other.community) && Objects.equals(projectId, other.projectId);
+	}
+
+	@Override
+	public String toString() {
+		return String.format("CommunityProjectPK [community=%s, projectId=%s]", community, projectId);
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbSubCommunity.java

@@ -0,0 +1,122 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.Table;
+
+import org.hibernate.annotations.Type;
+import org.hibernate.annotations.TypeDef;
+import org.hibernate.annotations.TypeDefs;
+
+import com.vladmihalcea.hibernate.type.array.StringArrayType;
+import com.vladmihalcea.hibernate.type.json.JsonBinaryType;
+import com.vladmihalcea.hibernate.type.json.JsonStringType;
+
+import eu.dnetlib.openaire.exporter.model.context.Param;
+
+@Entity
+@Table(name = "community_subs")
+@TypeDefs({
+	@TypeDef(name = "string-array", typeClass = StringArrayType.class),
+	@TypeDef(name = "json", typeClass = JsonStringType.class),
+	@TypeDef(name = "jsonb", typeClass = JsonBinaryType.class)
+})
+public class DbSubCommunity implements Serializable {
+
+	private static final long serialVersionUID = 7104936574383307358L;
+
+	@Id
+	@Column(name = "sub_id")
+	private String id;
+
+	@Column(name = "community")
+	private String community;
+
+	@Column(name = "label")
+	private String label;
+
+	@Column(name = "category")
+	private String category;
+
+	@Type(type = "jsonb")
+	@Column(name = "params")
+	private List<Param> params = new ArrayList<>();
+
+	@Column(name = "parent")
+	private String parent;
+
+	@Column(name = "claim")
+	private boolean claim = false;
+
+	@Column(name = "browsable")
+	private boolean browsable = false;
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(final String id) {
+		this.id = id;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getLabel() {
+		return label;
+	}
+
+	public void setLabel(final String label) {
+		this.label = label;
+	}
+
+	public String getCategory() {
+		return category;
+	}
+
+	public void setCategory(final String category) {
+		this.category = category;
+	}
+
+	public List<Param> getParams() {
+		return params;
+	}
+
+	public void setParams(final List<Param> params) {
+		this.params = params;
+	}
+
+	public String getParent() {
+		return parent;
+	}
+
+	public void setParent(final String parent) {
+		this.parent = parent;
+	}
+
+	public boolean isClaim() {
+		return claim;
+	}
+
+	public void setClaim(final boolean claim) {
+		this.claim = claim;
+	}
+
+	public boolean isBrowsable() {
+		return browsable;
+	}
+
+	public void setBrowsable(final boolean browsable) {
+		this.browsable = browsable;
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbSupportOrg.java

@@ -0,0 +1,64 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Table;
+
+@Entity
+@Table(name = "community_support_orgs")
+@IdClass(DbSupportOrgPK.class)
+public class DbSupportOrg implements Serializable {
+
+	private static final long serialVersionUID = 1308759097276753411L;
+
+	@Id
+	@Column(name = "community")
+	private String community;
+
+	@Id
+	@Column(name = "org_name")
+	private String orgName;
+
+	@Column(name = "org_url")
+	private String orgUrl;
+
+	@Column(name = "org_logourl")
+	private String orgLogoUrl;
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getOrgName() {
+		return orgName;
+	}
+
+	public void setOrgName(final String orgName) {
+		this.orgName = orgName;
+	}
+
+	public String getOrgUrl() {
+		return orgUrl;
+	}
+
+	public void setOrgUrl(final String orgUrl) {
+		this.orgUrl = orgUrl;
+	}
+
+	public String getOrgLogoUrl() {
+		return orgLogoUrl;
+	}
+
+	public void setOrgLogoUrl(final String orgLogoUrl) {
+		this.orgLogoUrl = orgLogoUrl;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DbSupportOrgPK.java

@@ -0,0 +1,55 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class DbSupportOrgPK implements Serializable {
+
+	private static final long serialVersionUID = -4117154543803798310L;
+
+	private String community;
+
+	private String orgName;
+
+	public DbSupportOrgPK() {}
+
+	public DbSupportOrgPK(final String community, final String orgName) {
+		this.community = community;
+		this.orgName = orgName;
+	}
+
+	public String getCommunity() {
+		return community;
+	}
+
+	public void setCommunity(final String community) {
+		this.community = community;
+	}
+
+	public String getOrgName() {
+		return orgName;
+	}
+
+	public void setOrgName(final String orgName) {
+		this.orgName = orgName;
+	}
+
+	@Override
+	public int hashCode() {
+		return Objects.hash(community, orgName);
+	}
+
+	@Override
+	public boolean equals(final Object obj) {
+		if (this == obj) { return true; }
+		if (!(obj instanceof DbSupportOrgPK)) { return false; }
+		final DbSupportOrgPK other = (DbSupportOrgPK) obj;
+		return Objects.equals(community, other.community) && Objects.equals(orgName, other.orgName);
+	}
+
+	@Override
+	public String toString() {
+		return String.format("CommunitySupportOrgPK [community=%s, orgName=%s]", community, orgName);
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/model/DepositionInfo.java

@@ -0,0 +1,39 @@
+package eu.dnetlib.openaire.community.model;
+
+import java.io.Serializable;
+
+public class DepositionInfo implements Serializable {
+
+	private static final long serialVersionUID = 7538663287451167904L;
+
+	private String openaireId;
+
+	private Boolean deposit;
+
+	private String message;
+
+	public String getOpenaireId() {
+		return openaireId;
+	}
+
+	public void setOpenaireId(final String openaireId) {
+		this.openaireId = openaireId;
+	}
+
+	public Boolean getDeposit() {
+		return deposit;
+	}
+
+	public void setDeposit(final Boolean deposit) {
+		this.deposit = deposit;
+	}
+
+	public String getMessage() {
+		return message;
+	}
+
+	public void setMessage(final String message) {
+		this.message = message;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbCommunityRepository.java

@@ -0,0 +1,17 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.jpa.repository.JpaRepository;
+import org.springframework.data.jpa.repository.Query;
+
+import eu.dnetlib.openaire.community.model.DbCommunity;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbCommunityRepository extends JpaRepository<DbCommunity, String> {
+
+	@Query(value = "select id from communities where ?1 = ANY(array_append(other_zenodo_communities, main_zenodo_community))", nativeQuery = true)
+	List<String> findByZenodoId(String zenodoId);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbDatasourceRepository.java

@@ -0,0 +1,20 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import eu.dnetlib.openaire.community.model.DbDatasource;
+import eu.dnetlib.openaire.community.model.DbDatasourcePK;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbDatasourceRepository extends JpaRepository<DbDatasource, DbDatasourcePK> {
+
+	List<DbDatasource> findByCommunity(String community);
+
+	List<DbDatasource> findByCommunityAndDeposit(String community, boolean deposit);
+
+	void deleteByCommunity(String id);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbOrganizationRepository.java

@@ -0,0 +1,18 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import eu.dnetlib.openaire.community.model.DbOrganization;
+import eu.dnetlib.openaire.community.model.DbOrganizationPK;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbOrganizationRepository extends JpaRepository<DbOrganization, DbOrganizationPK> {
+
+	List<DbOrganization> findByCommunity(String community);
+
+	void deleteByCommunity(String id);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbProjectRepository.java

@@ -0,0 +1,25 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.domain.Page;
+import org.springframework.data.domain.Pageable;
+import org.springframework.data.jpa.repository.JpaRepository;
+import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
+import org.springframework.data.jpa.repository.Query;
+
+import eu.dnetlib.openaire.community.model.DbProject;
+import eu.dnetlib.openaire.community.model.DbProjectPK;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbProjectRepository extends JpaRepository<DbProject, DbProjectPK>, JpaSpecificationExecutor<DbProject> {
+
+	Page<DbProject> findByCommunity(String community, Pageable page);
+
+	void deleteByCommunity(String id);
+
+	@Query(value = "select distinct project_funder from community_projects where community = ?1 order by project_funder", nativeQuery = true)
+	List<String> findFundersByCommunity(String id);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbSubCommunityRepository.java

@@ -0,0 +1,19 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import eu.dnetlib.openaire.community.model.DbSubCommunity;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbSubCommunityRepository extends JpaRepository<DbSubCommunity, String> {
+
+	List<DbSubCommunity> findByCommunity(String community);
+
+	List<DbSubCommunity> findByCommunityAndParent(String community, String parent);
+
+	void deleteByCommunity(String id);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/repository/DbSupportOrgRepository.java

@@ -0,0 +1,18 @@
+package eu.dnetlib.openaire.community.repository;
+
+import java.util.List;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import eu.dnetlib.openaire.community.model.DbSupportOrg;
+import eu.dnetlib.openaire.community.model.DbSupportOrgPK;
+
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
+public interface DbSupportOrgRepository extends JpaRepository<DbSupportOrg, DbSupportOrgPK> {
+
+	List<DbSupportOrg> findByCommunity(String community);
+
+	void deleteByCommunity(String id);
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/selectioncriteria/SelectionCriteria.java

@@ -1,33 +0,0 @@
-package eu.dnetlib.openaire.community.selectioncriteria;
-
-import java.io.Serializable;
-import java.util.List;
-
-import com.fasterxml.jackson.annotation.JsonAutoDetect;
-import com.google.gson.Gson;
-
-@JsonAutoDetect
-public class SelectionCriteria implements Serializable {
-
-	/**
-	 *
-	 */
-	private static final long serialVersionUID = 4303936216579280542L;
-
-	private List<Constraints> criteria;
-
-	public SelectionCriteria() {}
-
-	public List<Constraints> getCriteria() {
-		return criteria;
-	}
-
-	public void setCriteria(final List<Constraints> criteria) {
-		this.criteria = criteria;
-	}
-
-	public static SelectionCriteria fromJson(final String json) {
-		return new Gson().fromJson(json, SelectionCriteria.class);
-
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/utils/CommunityClaimTypeConverter.java

@@ -0,0 +1,29 @@
+package eu.dnetlib.openaire.community.utils;
+
+import javax.persistence.AttributeConverter;
+
+import org.apache.commons.lang3.StringUtils;
+
+import eu.dnetlib.openaire.exporter.model.community.CommunityClaimType;
+
+public class CommunityClaimTypeConverter implements AttributeConverter<CommunityClaimType, String> {
+
+	@Override
+	public String convertToDatabaseColumn(final CommunityClaimType attribute) {
+		if (attribute == null) {
+			return null;
+		} else {
+			return attribute.getDescription();
+		}
+	}
+
+	@Override
+	public CommunityClaimType convertToEntityAttribute(final String dbData) {
+		if (StringUtils.isBlank(dbData)) {
+			return null;
+		} else {
+			return CommunityClaimType.fromDescription(dbData);
+		}
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/utils/CommunityMappingUtils.java

@@ -0,0 +1,315 @@
+package eu.dnetlib.openaire.community.utils;
+
+import java.text.ParseException;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.ZoneId;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.List;
+
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.time.DateUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import com.google.common.collect.Lists;
+
+import eu.dnetlib.openaire.community.importer.CommunityImporterService;
+import eu.dnetlib.openaire.community.model.DbCommunity;
+import eu.dnetlib.openaire.community.model.DbDatasource;
+import eu.dnetlib.openaire.community.model.DbProject;
+import eu.dnetlib.openaire.community.model.DbSubCommunity;
+import eu.dnetlib.openaire.community.model.DbSupportOrg;
+import eu.dnetlib.openaire.exporter.model.community.CommunityContentprovider;
+import eu.dnetlib.openaire.exporter.model.community.CommunityDetails;
+import eu.dnetlib.openaire.exporter.model.community.CommunityOrganization;
+import eu.dnetlib.openaire.exporter.model.community.CommunityProject;
+import eu.dnetlib.openaire.exporter.model.community.CommunitySummary;
+import eu.dnetlib.openaire.exporter.model.community.CommunityWritableProperties;
+import eu.dnetlib.openaire.exporter.model.community.SubCommunity;
+import eu.dnetlib.openaire.exporter.model.context.IISConfigurationEntry;
+
+public class CommunityMappingUtils {
+
+	public final static String PIPE_SEPARATOR = "||";
+
+	private static final List<String> DATE_PATTERN = Lists.newArrayList("yyyy-MM-dd'T'hh:mm:ss", "yyyy-MM-dd'T'hh:mm:ssXXX", "yyyy-MM-dd'T'hh:mm:ss+00:00");
+
+	private static final Log log = LogFactory.getLog(CommunityMappingUtils.class);
+
+	public static CommunitySummary toCommunitySummary(final DbCommunity c) {
+		final CommunitySummary summary = new CommunitySummary();
+		populateSummary(summary, c);
+		return summary;
+	}
+
+	public static DbCommunity toCommunity(final CommunityDetails details) {
+		final DbCommunity c = new DbCommunity();
+		c.setId(details.getId());
+		c.setName(details.getName());
+		c.setShortName(details.getShortName());
+		c.setDisplayName(details.getDisplayName());
+		c.setDisplayShortName(details.getDisplayShortName());
+		c.setDescription(details.getDescription());
+		c.setStatus(details.getStatus());
+		c.setLogoUrl(details.getLogoUrl());
+		c.setMembership(details.getMembership());
+		c.setType(details.getType());
+		c.setClaim(details.getClaim());
+		c.setSubjects(toStringArray(details.getSubjects()));
+		c.setFos(toStringArray(details.getFos()));
+		c.setSdg(toStringArray(details.getSdg()));
+		c.setAdvancedConstraints(details.getAdvancedConstraints());
+		c.setRemoveConstraints(details.getRemoveConstraints());
+		c.setMainZenodoCommunity(details.getZenodoCommunity());
+		c.setOtherZenodoCommunities(toStringArray(details.getOtherZenodoCommunities()));
+		c.setSuggestedAcknowledgements(toStringArray(details.getSuggestedAcknowledgements()));
+		c.setPlan(details.getPlan());
+		c.setCreationDate(ObjectUtils.firstNonNull(details.getCreationDate(), LocalDateTime.now()));
+		c.setLastUpdateDate(LocalDateTime.now());
+		return c;
+	}
+
+	public static void populateCommunity(final DbCommunity c, final CommunityWritableProperties details) {
+		if (StringUtils.isNotBlank(details.getName())) {
+			c.setName(details.getName());
+		}
+		if (StringUtils.isNotBlank(details.getShortName())) {
+			c.setShortName(details.getShortName());
+		}
+		if (StringUtils.isNotBlank(details.getDisplayName())) {
+			c.setDisplayName(details.getDisplayName());
+		}
+		if (StringUtils.isNotBlank(details.getDisplayShortName())) {
+			c.setDisplayShortName(details.getDisplayShortName());
+		}
+		if (StringUtils.isNotBlank(details.getDescription())) {
+			c.setDescription(details.getDescription());
+		}
+		if (details.getStatus() != null) {
+			c.setStatus(details.getStatus());
+		}
+		if (details.getMembership() != null) {
+			c.setMembership(details.getMembership());
+		}
+		if (details.getType() != null) {
+			c.setType(details.getType());
+		}
+		if (details.getClaim() != null) {
+			c.setClaim(details.getClaim());
+		}
+		if (StringUtils.isNotBlank(details.getLogoUrl())) {
+			c.setLogoUrl(details.getLogoUrl());
+		}
+		if (details.getFos() != null) {
+			c.setFos(toStringArray(details.getFos()));
+		}
+		if (details.getSdg() != null) {
+			c.setSdg(toStringArray(details.getSdg()));
+		}
+		if (details.getSubjects() != null) {
+			c.setSubjects(toStringArray(details.getSubjects()));
+		}
+		if (details.getAdvancedConstraints() != null) {
+			c.setAdvancedConstraints(details.getAdvancedConstraints());
+		}
+		if (details.getRemoveConstraints() != null) {
+			c.setRemoveConstraints(details.getRemoveConstraints());
+		}
+		if (StringUtils.isNotBlank(details.getMainZenodoCommunity())) {
+			c.setMainZenodoCommunity(details.getMainZenodoCommunity());
+		}
+		if (details.getOtherZenodoCommunities() != null) {
+			c.setOtherZenodoCommunities(toStringArray(details.getOtherZenodoCommunities()));
+		}
+		if (details.getPlan() != null) {
+			c.setPlan(details.getPlan());
+		}
+
+		c.setLastUpdateDate(LocalDateTime.now());
+	}
+
+	public static CommunityDetails toCommunityDetails(final DbCommunity c) {
+		final CommunityDetails details = new CommunityDetails();
+		populateSummary(details, c);
+		details.setAdvancedConstraints(c.getAdvancedConstraints());
+		details.setRemoveConstraints(c.getRemoveConstraints());
+		details.setFos(Arrays.asList(c.getFos()));
+		details.setSdg(Arrays.asList(c.getSdg()));
+		details.setSubjects(Arrays.asList(c.getSubjects()));
+		details.setOtherZenodoCommunities(Arrays.asList(c.getOtherZenodoCommunities()));
+		details.setSuggestedAcknowledgements(Arrays.asList(c.getSuggestedAcknowledgements()));
+		return details;
+	}
+
+	private static void populateSummary(final CommunitySummary summary, final DbCommunity c) {
+		summary.setId(c.getId());
+		summary.setName(c.getName());
+		summary.setShortName(c.getShortName());
+		summary.setDisplayName(c.getDisplayName());
+		summary.setDisplayShortName(c.getDisplayShortName());
+		summary.setLastUpdateDate(c.getLastUpdateDate());
+		summary.setCreationDate(c.getCreationDate());
+		summary.setQueryId(c.getId() + PIPE_SEPARATOR + c.getShortName());
+		summary.setType(c.getType());
+		summary.setDescription(c.getDescription());
+		summary.setLogoUrl(c.getLogoUrl());
+		summary.setStatus(c.getStatus());
+		summary.setClaim(c.getClaim());
+		summary.setMembership(c.getMembership());
+		summary.setZenodoCommunity(c.getMainZenodoCommunity());
+		summary.setPlan(c.getPlan());
+	}
+
+	public static CommunityProject toCommunityProject(final DbProject dbEntry) {
+		final CommunityProject cp = new CommunityProject();
+		cp.setCommunityId(dbEntry.getCommunity());
+		cp.setOpenaireId(dbEntry.getProjectId());
+		cp.setName(dbEntry.getProjectName());
+		cp.setAcronym(dbEntry.getProjectAcronym());
+		cp.setFunder(dbEntry.getProjectFunder());
+		cp.setGrantId(dbEntry.getProjectCode());
+		cp.setAvailableSince(dbEntry.getAvailableSince());
+		return cp;
+	}
+
+	public static DbProject toDbProject(final String id, final CommunityProject project) {
+		final DbProject p = new DbProject();
+		p.setCommunity(id);
+		p.setProjectId(project.getOpenaireId());
+		p.setProjectName(project.getName());
+		p.setProjectAcronym(project.getAcronym());
+		p.setProjectCode(project.getGrantId());
+		p.setProjectFunder(project.getFunder());
+		if (project.getAvailableSince() != null) {
+			p.setAvailableSince(project.getAvailableSince());
+		} else {
+			p.setAvailableSince(LocalDate.now());
+		}
+		return p;
+	}
+
+	public static CommunityContentprovider toCommunityContentprovider(final DbDatasource dbEntry) {
+		final CommunityContentprovider ccp = new CommunityContentprovider();
+		ccp.setCommunityId(dbEntry.getCommunity());
+		ccp.setOpenaireId(dbEntry.getDsId());
+		ccp.setName(dbEntry.getDsName());
+		ccp.setOfficialname(dbEntry.getDsOfficialName());
+		ccp.setSelectioncriteria(dbEntry.getConstraints());
+		ccp.setEnabled(dbEntry.getEnabled() != null ? dbEntry.getEnabled() : true);
+		ccp.setDeposit(dbEntry.getDeposit() != null ? dbEntry.getDeposit() : false);
+		ccp.setMessage(dbEntry.getMessage());
+		return ccp;
+	}
+
+	public static DbDatasource toDbDatasource(final String id, final CommunityContentprovider provider) {
+		final DbDatasource ds = new DbDatasource();
+		ds.setCommunity(id);
+		ds.setDsId(provider.getOpenaireId());
+		ds.setDsName(provider.getName());
+		ds.setDsOfficialName(provider.getOfficialname());
+		ds.setConstraints(provider.getSelectioncriteria());
+		ds.setEnabled(provider.isEnabled());
+		ds.setDeposit(provider.getDeposit() != null ? provider.getDeposit() : false);
+		ds.setMessage(provider.getMessage());
+		return ds;
+	}
+
+	public static CommunityOrganization toCommunityOrganization(final DbSupportOrg dbEntry) {
+		final CommunityOrganization co = new CommunityOrganization();
+		co.setCommunityId(dbEntry.getCommunity());
+		co.setName(dbEntry.getOrgName());
+		co.setWebsite_url(dbEntry.getOrgUrl());
+		co.setLogo_url(dbEntry.getOrgLogoUrl());
+		return co;
+	}
+
+	public static DbSupportOrg toDbSupportOrg(final String id, final CommunityOrganization org) {
+		final DbSupportOrg dbo = new DbSupportOrg();
+		dbo.setCommunity(id);
+		dbo.setOrgName(org.getName());
+		dbo.setOrgUrl(org.getWebsite_url());
+		dbo.setOrgLogoUrl(org.getLogo_url());
+		return dbo;
+	}
+
+	public static DbSubCommunity toDbSubCommunity(final String id, final SubCommunity sub) {
+		final DbSubCommunity dbsc = new DbSubCommunity();
+		dbsc.setCommunity(id);
+		dbsc.setId(sub.getSubCommunityId());
+		dbsc.setCategory(sub.getCategory());
+		dbsc.setLabel(sub.getLabel());
+		dbsc.setParams(sub.getParams());
+		dbsc.setParent(sub.getParent());
+		dbsc.setClaim(sub.isClaim());
+		dbsc.setBrowsable(sub.isBrowsable());
+		return dbsc;
+	}
+
+	public static SubCommunity toSubCommunity(final DbSubCommunity sub) {
+		final SubCommunity sc = new SubCommunity();
+		sc.setSubCommunityId(sub.getId());
+		sc.setCategory(sub.getCategory());
+		sc.setCommunityId(sub.getCommunity());
+		sc.setLabel(sub.getLabel());
+		sc.setParams(sub.getParams());
+		sc.setParent(sub.getParent());
+		sc.setClaim(sub.isClaim());
+		sc.setBrowsable(sub.isBrowsable());
+		return sc;
+	}
+
+	public static LocalDateTime asLocalDateTime(final String s) {
+		if (StringUtils.isBlank(s)) { return null; }
+
+		for (final String pattern : DATE_PATTERN) {
+			try {
+				final Date res = DateUtils.parseDate(s, pattern);
+				if (res != null) { return asLocalDateTime(res); }
+			} catch (final ParseException e) {}
+		}
+		log.warn("Invalid Date: " + s);
+		return null;
+	}
+
+	public static LocalDateTime asLocalDateTime(final Date date) {
+		return date.toInstant()
+				.atZone(ZoneId.systemDefault())
+				.toLocalDateTime();
+	}
+
+	private static String[] toStringArray(final List<String> list) {
+		return list != null ? list.toArray(new String[list.size()]) : new String[0];
+	}
+
+	public static IISConfigurationEntry asIISConfigurationEntry(final DbCommunity c) {
+		final IISConfigurationEntry conf = new IISConfigurationEntry();
+		conf.setId(c.getId());
+		conf.setLabel(c.getName());
+		conf.addParams(CommunityImporterService.CSUMMARY_DESCRIPTION, c.getDescription());
+		conf.addParams(CommunityImporterService.CSUMMARY_LOGOURL, c.getLogoUrl());
+		conf.addParams(CommunityImporterService.CSUMMARY_STATUS, c.getStatus().toString());
+		conf.addParams(CommunityImporterService.CSUMMARY_NAME, c.getName());
+		conf.addParams(CommunityImporterService.CSUMMARY_ZENODOC, c.getMainZenodoCommunity());
+		conf.addParams(CommunityImporterService.CPROFILE_SUBJECT, c.getSubjects());
+		conf.addParams(CommunityImporterService.CPROFILE_FOS, c.getFos());
+		conf.addParams(CommunityImporterService.CPROFILE_SDG, c.getSdg());
+		conf.addParams(CommunityImporterService.CPROFILE_ADVANCED_CONSTRAINT, c.getAdvancedConstraints() != null ? c.getAdvancedConstraints().toJson() : null);
+		conf.addParams(CommunityImporterService.CPROFILE_REMOVE_CONSTRAINT, c.getRemoveConstraints() != null ? c.getRemoveConstraints().toJson() : null);
+		conf.addParams(CommunityImporterService.CPROFILE_SUGGESTED_ACKNOWLEDGEMENT, c.getSuggestedAcknowledgements());
+		conf.addParams(CommunityImporterService.CPROFILE_CREATIONDATE, c.getCreationDate() != null ? c.getCreationDate().toString() : null);
+		return conf;
+	}
+
+	public static IISConfigurationEntry asIISConfigurationEntry(final DbSubCommunity subc) {
+		final IISConfigurationEntry conf = new IISConfigurationEntry();
+		conf.setId(subc.getId());
+		conf.setLabel(subc.getLabel());
+		if (subc.getParams() != null) {
+			conf.getParams().addAll(subc.getParams());
+		}
+		return conf;
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/community/utils/CommunityMembershipTypeConverter.java

@@ -0,0 +1,29 @@
+package eu.dnetlib.openaire.community.utils;
+
+import javax.persistence.AttributeConverter;
+
+import org.apache.commons.lang3.StringUtils;
+
+import eu.dnetlib.openaire.exporter.model.community.CommunityMembershipType;
+
+public class CommunityMembershipTypeConverter implements AttributeConverter<CommunityMembershipType, String> {
+
+	@Override
+	public String convertToDatabaseColumn(final CommunityMembershipType attribute) {
+		if (attribute == null) {
+			return null;
+		} else {
+			return attribute.getDescription();
+		}
+	}
+
+	@Override
+	public CommunityMembershipType convertToEntityAttribute(final String dbData) {
+		if (StringUtils.isBlank(dbData)) {
+			return null;
+		} else {
+			return CommunityMembershipType.fromDescription(dbData);
+		}
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/context/ContextApiController.java

@@ -1,8 +1,13 @@
 package eu.dnetlib.openaire.context;
 
 import java.util.List;
-import java.util.Optional;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
 
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
 import org.springframework.web.bind.annotation.CrossOrigin;
@@ -12,6 +17,16 @@ import org.springframework.web.bind.annotation.RequestMethod;
 import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
 
+import eu.dnetlib.openaire.common.AbstractExporterController;
+import eu.dnetlib.openaire.community.CommunityService;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.exceptions.ResourceNotFoundException;
+import eu.dnetlib.openaire.exporter.model.community.CommunityType;
+import eu.dnetlib.openaire.exporter.model.community.SubCommunity;
+import eu.dnetlib.openaire.exporter.model.context.CategorySummary;
+import eu.dnetlib.openaire.exporter.model.context.ConceptSummary;
+import eu.dnetlib.openaire.exporter.model.context.ContextSummary;
+import eu.dnetlib.openaire.exporter.model.context.IISConfigurationEntry;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.responses.ApiResponses;
@@ -21,12 +36,14 @@ import io.swagger.v3.oas.annotations.tags.Tag;
 @CrossOrigin(origins = {
 	"*"
 })
-@ConditionalOnProperty(value = "openaire.exporter.enable.context", havingValue = "true")
+@ConditionalOnProperty(value = "openaire.exporter.enable.community", havingValue = "true")
 @Tag(name = "OpenAIRE Context API", description = "the OpenAIRE Context API")
-public class ContextApiController {
+public class ContextApiController extends AbstractExporterController {
 
 	@Autowired
-	private ContextApiCore contextApiCore;
+	private CommunityService communityService;
+
+	private static final Log log = LogFactory.getLog(ContextApiController.class);
 
 	@RequestMapping(value = "/contexts", produces = {
 		"application/json"
@@ -34,10 +51,28 @@ public class ContextApiController {
 	@Operation(summary = "list brief information about all the context profiles", description = "list brief information about all the context profiles")
 	@ApiResponses(value = {
 		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "404", description = "not found"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public List<ContextSummary> listContexts(@RequestParam(required = false, defaultValue = "") final List<String> type) throws ContextException {
-		return contextApiCore.listContexts(type);
+	public List<ContextSummary> listContexts(@RequestParam(required = false) final Set<CommunityType> type) throws CommunityException {
+		try {
+			return communityService.listCommunities()
+				.stream()
+				.filter(c -> type == null || type.contains(c.getType()))
+				.map(c -> {
+					final ContextSummary ctx = new ContextSummary();
+					ctx.setId(c.getId());
+					ctx.setLabel(c.getName());
+					ctx.setStatus(c.getStatus().toString());
+					ctx.setType(c.getType().toString());
+					return ctx;
+				})
+				.collect(Collectors.toList());
+		} catch (final ResourceNotFoundException e) {
+			throw e;
+		} catch (final Throwable e) {
+			throw new CommunityException(e);
+		}
 	}
 
 	@RequestMapping(value = "/context/{contextId}", produces = {
@@ -46,14 +81,52 @@ public class ContextApiController {
 	@Operation(summary = "list the categories defined within a context", description = "list the categories defined within a context")
 	@ApiResponses(value = {
 		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "404", description = "not found"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
 	public List<CategorySummary> listCategories(
 		@PathVariable final String contextId,
-		@RequestParam(required = false, defaultValue = "false") final Boolean all) throws ContextException {
+		@RequestParam(required = false, defaultValue = "false") final boolean all) throws CommunityException {
 
-		final Boolean allFilter = Optional.ofNullable(all).orElse(false);
-		return contextApiCore.listCategories(contextId, allFilter);
+		try {
+			return communityService.getSubCommunities(contextId)
+				.stream()
+				.filter(sc -> all || sc.isClaim())
+				.map(sc -> {
+					final String[] parts = StringUtils.split(sc.getSubCommunityId(), "::");
+					if (parts.length < 3) { throw new RuntimeException("Invalid conceptId (It should have 3 (or more) parts): " + sc.getSubCommunityId()); }
+					final CategorySummary cat = new CategorySummary();
+					cat.setId(parts[0] + "::" + parts[1]);
+					cat.setLabel(sc.getCategory());
+					cat.setHasConcept(true);
+					return cat;
+				})
+				.distinct()
+				.collect(Collectors.toList());
+		} catch (final ResourceNotFoundException e) {
+			throw e;
+		} catch (final Throwable e) {
+			throw new CommunityException(e);
+		}
+	}
+
+	@RequestMapping(value = "/context/iis/conf/{contextId}", produces = {
+		"application/json"
+	}, method = RequestMethod.GET)
+	@Operation(summary = "return a list of entries for IIS", description = "return a list of entries for IIS")
+	@ApiResponses(value = {
+		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "404", description = "not found"),
+		@ApiResponse(responseCode = "500", description = "unexpected error")
+	})
+	public List<IISConfigurationEntry> getIISConfiguration(@PathVariable final String contextId) throws CommunityException {
+		try {
+			return communityService.getIISConfiguration(contextId);
+		} catch (final ResourceNotFoundException e) {
+			throw e;
+		} catch (final Throwable e) {
+			throw new CommunityException(e);
+		}
 	}
 
 	@RequestMapping(value = "/context/category/{categoryId}", produces = {
@@ -62,14 +135,30 @@ public class ContextApiController {
 	@Operation(summary = "list the concepts defined within a category", description = "list the concepts defined within a category")
 	@ApiResponses(value = {
 		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "404", description = "not found"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
 	public List<ConceptSummary> listConcepts(
 		@PathVariable final String categoryId,
-		@RequestParam(required = false, defaultValue = "false") final Boolean all) throws ContextException {
+		@RequestParam(required = false, defaultValue = "false") final boolean all) throws CommunityException {
+		try {
+			final String[] parts = StringUtils.split(categoryId, "::");
+			if (parts.length != 2) {
+				log.error("Invalid category id (it should have 2 parts): " + categoryId);
+				throw new CommunityException("Invalid category id (it should have 2 parts): " + categoryId);
+			}
+
+			final String contextId = parts[0];
+
+			final List<SubCommunity> list = findSubCommunities(categoryId + "::", all, contextId);
+
+			return processSubCommunities(null, list);
+		} catch (final ResourceNotFoundException e) {
+			throw e;
+		} catch (final Throwable e) {
+			throw new CommunityException(e);
+		}
 
-		final Boolean allFilter = Optional.ofNullable(all).orElse(false);
-		return contextApiCore.listConcepts(categoryId, allFilter);
 	}
 
 	@RequestMapping(value = "/context/category/concept/{conceptId}", produces = {
@@ -78,14 +167,53 @@ public class ContextApiController {
 	@Operation(summary = "list the concepts defined within a category", description = "list the concepts defined within a category")
 	@ApiResponses(value = {
 		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "404", description = "not found"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
 	public List<ConceptSummary> listSubConcepts(
 		@PathVariable final String conceptId,
-		@RequestParam(required = false, defaultValue = "false") final Boolean all) throws ContextException {
+		@RequestParam(required = false, defaultValue = "false") final boolean all) throws CommunityException {
+		try {
+			final String[] parts = StringUtils.split(conceptId, "::");
+			if (parts.length < 3) {
+				log.error("Invalid concept id (it should have 3 (or more) parts): " + conceptId);
+				throw new CommunityException("Invalid concept id (it should have 3 (or more) parts): " + conceptId);
+			}
 
-		final Boolean allFilter = Optional.ofNullable(all).orElse(false);
-		return contextApiCore.listSubConcepts(conceptId, allFilter);
+			final String contextId = parts[0];
+
+			final List<SubCommunity> list = findSubCommunities(conceptId + "::", all, contextId);
+
+			return processSubCommunities(conceptId, list);
+		} catch (final ResourceNotFoundException e) {
+			throw e;
+		} catch (final Throwable e) {
+			throw new CommunityException(e);
+		}
+
+	}
+
+	private List<SubCommunity> findSubCommunities(final String prefix, final boolean all, final String contextId) throws CommunityException {
+		return communityService.getSubCommunities(contextId)
+			.stream()
+			.filter(sc -> all || sc.isClaim())
+			.filter(sc -> sc.getSubCommunityId().startsWith(prefix))
+			.collect(Collectors.toList());
+	}
+
+	private List<ConceptSummary> processSubCommunities(final String parent, final List<SubCommunity> list) {
+		return list.stream()
+			.filter(sc -> Objects.equals(sc.getParent(), parent))
+			.map(sc -> {
+				final List<ConceptSummary> childs = processSubCommunities(sc.getSubCommunityId(), list);
+				final ConceptSummary concept = new ConceptSummary();
+				concept.setId(sc.getSubCommunityId());
+				concept.setLabel(sc.getLabel());
+				concept.setHasSubConcept(!childs.isEmpty());
+				concept.setConcept(childs);
+				return concept;
+			})
+			.collect(Collectors.toList());
 	}
 
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/context/ContextApiCore.java

@@ -8,16 +8,22 @@ import java.util.stream.Stream;
 
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
-import org.springframework.stereotype.Component;
 
 import com.google.common.base.Splitter;
 import com.google.common.collect.Lists;
 
 import eu.dnetlib.openaire.common.ISClient;
+import eu.dnetlib.openaire.exporter.exceptions.CommunityException;
+import eu.dnetlib.openaire.exporter.model.context.Category;
+import eu.dnetlib.openaire.exporter.model.context.CategorySummary;
+import eu.dnetlib.openaire.exporter.model.context.Concept;
+import eu.dnetlib.openaire.exporter.model.context.ConceptSummary;
+import eu.dnetlib.openaire.exporter.model.context.Context;
+import eu.dnetlib.openaire.exporter.model.context.ContextSummary;
 
-@Component
-@ConditionalOnProperty(value = "openaire.exporter.enable.context", havingValue = "true")
+// @Component
+// @ConditionalOnProperty(value = "openaire.exporter.enable.context", havingValue = "true")
+@Deprecated
 public class ContextApiCore {
 
 	private static final String SEPARATOR = "::";
@@ -25,18 +31,24 @@ public class ContextApiCore {
 	@Autowired
 	private ISClient isClient;
 
-	public List<ContextSummary> listContexts(final List<String> type) throws ContextException {
+	public List<ContextSummary> listContexts(final List<String> type) throws CommunityException {
+
 		return getContextMap(type).values()
 			.stream()
 			.map(c -> new ContextSummary()
 				.setId(c.getId())
 				.setType(c.getType())
 				.setLabel(c.getLabel())
-				.setStatus(c.getParams().containsKey("status") ? c.getParams().get("status").get(0).getValue() : ""))
+				.setStatus(c.getParams()
+					.stream()
+					.filter(p -> p.getName().equals("status"))
+					.map(p -> p.getValue())
+					.findFirst()
+					.orElse("")))
 			.collect(Collectors.toList());
 	}
 
-	public List<CategorySummary> listCategories(final String contextId, final Boolean all) throws ContextException {
+	public List<CategorySummary> listCategories(final String contextId, final Boolean all) throws CommunityException {
 		final Stream<Category> categories = getContextMap().get(contextId).getCategories().values().stream();
 		return all ? asCategorySummaries(categories) : asCategorySummaries(categories.filter(Category::isClaim));
 	}
@@ -50,7 +62,7 @@ public class ContextApiCore {
 			.collect(Collectors.toList());
 	}
 
-	public List<ConceptSummary> listConcepts(final String categoryId, final Boolean all) throws ContextException {
+	public List<ConceptSummary> listConcepts(final String categoryId, final Boolean all) throws CommunityException {
 		final String contextId = StringUtils.substringBefore(categoryId, SEPARATOR);
 		final Stream<Concept> concepts = getContextMap().get(contextId)
 			.getCategories()
@@ -70,9 +82,9 @@ public class ContextApiCore {
 			.collect(Collectors.toList());
 	}
 
-	public List<ConceptSummary> listSubConcepts(final String conceptId, final Boolean all) throws ContextException {
+	public List<ConceptSummary> listSubConcepts(final String conceptId, final Boolean all) throws CommunityException {
 		final List<String> ids = Splitter.on(SEPARATOR).splitToList(conceptId);
-		if (ids.size() < 3) { throw new ContextException(""); }
+		if (ids.size() < 3) { throw new CommunityException(""); }
 
 		final String contextId = ids.get(0);
 		final String categoryId = contextId + SEPARATOR + ids.get(1);
@@ -98,15 +110,15 @@ public class ContextApiCore {
 			.collect(Collectors.toList());
 	}
 
-	private Map<String, Context> getContextMap() throws ContextException {
+	private Map<String, Context> getContextMap() throws CommunityException {
 		return getContextMap(Lists.newArrayList());
 	}
 
-	private Map<String, Context> getContextMap(final List<String> type) throws ContextException {
+	private Map<String, Context> getContextMap(final List<String> type) throws CommunityException {
 		try {
 			return isClient.getContextMap(type);
 		} catch (final IOException e) {
-			throw new ContextException(e);
+			throw new CommunityException(e);
 		}
 	}
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/context/ContextException.java

@@ -1,25 +0,0 @@
-package eu.dnetlib.openaire.context;
-
-import java.io.IOException;
-
-import org.springframework.http.HttpStatus;
-import org.springframework.web.bind.annotation.ResponseBody;
-import org.springframework.web.bind.annotation.ResponseStatus;
-
-@ResponseBody
-@ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
-public class ContextException extends Exception {
-
-	/**
-	 *
-	 */
-	private static final long serialVersionUID = -5489369676370127052L;
-
-	public ContextException(final String message) {
-		super(message);
-	}
-
-	public ContextException(final IOException e) {
-		super(e);
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/context/ContextMappingUtils.java

@@ -1,7 +1,6 @@
 package eu.dnetlib.openaire.context;
 
 import java.text.ParseException;
-import java.util.ArrayList;
 import java.util.Date;
 import java.util.List;
 import java.util.Map;
@@ -11,6 +10,8 @@ import java.util.stream.Collectors;
 import org.apache.commons.lang3.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.time.DateUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
 import org.dom4j.Document;
 import org.dom4j.DocumentException;
 import org.dom4j.DocumentHelper;
@@ -20,27 +21,37 @@ import org.dom4j.Node;
 import com.google.common.base.Functions;
 import com.google.common.collect.Lists;
 
-import eu.dnetlib.openaire.funders.domain.FunderDetails;
+import eu.dnetlib.openaire.exporter.model.context.Category;
+import eu.dnetlib.openaire.exporter.model.context.Concept;
+import eu.dnetlib.openaire.exporter.model.context.Context;
+import eu.dnetlib.openaire.exporter.model.context.Param;
+import eu.dnetlib.openaire.exporter.model.funders.FunderDetails;
 
+@Deprecated
 public class ContextMappingUtils {
 
 	private static final List<String> DATE_PATTERN = Lists.newArrayList("yyyy-MM-dd'T'hh:mm:ss", "yyyy-MM-dd'T'hh:mm:ssXXX", "yyyy-MM-dd'T'hh:mm:ss+00:00");
 
+	private static final Log log = LogFactory.getLog(ContextMappingUtils.class);
+
 	public static Context parseContext(final String s, final Queue<Throwable> errors) {
 		try {
 			final Document doc = DocumentHelper.parseText(s);
 			final Element eContext = (Element) doc.selectSingleNode("/RESOURCE_PROFILE/BODY/CONFIGURATION/context");
 			final String creationDate = eContext.valueOf("./param[./@name='creationdate']/text()");
+			final String otherDate = doc.valueOf("/RESOURCE_PROFILE/HEADER/DATE_OF_CREATION/@value");
 			final Context c = new Context()
 				.setId(eContext.attributeValue("id"))
 				.setLabel(eContext.attributeValue("label"))
 				.setType(eContext.attributeValue("type"))
-				.setLastUpdateDate(asDate(doc.valueOf("/RESOURCE_PROFILE/HEADER/DATE_OF_CREATION/@value")))
+				.setLastUpdateDate(asDate(otherDate))
 				.setParams(parseParams(eContext))
 				.setCategories(parseCategories(eContext));
 			// the creation date will be added in the param elements of the community profile. Funders may not have it, hence the check.
 			if (StringUtils.isNotBlank(creationDate)) {
 				c.setCreationDate(asDate(creationDate));
+			} else {
+				c.setCreationDate(asDate(otherDate));
 			}
 			return c;
 		} catch (final DocumentException e) {
@@ -50,11 +61,15 @@ public class ContextMappingUtils {
 	}
 
 	private static Date asDate(final String s) {
+		if (StringUtils.isBlank(s)) { return null; }
+
 		for (final String pattern : DATE_PATTERN) {
 			try {
-				return DateUtils.parseDate(s, pattern);
+				final Date res = DateUtils.parseDate(s, pattern);
+				if (res != null) { return res; }
 			} catch (final ParseException e) {}
 		}
+		log.warn("Invalid Date: " + s);
 		return null;
 	}
 
@@ -89,18 +104,12 @@ public class ContextMappingUtils {
 		return BooleanUtils.toBooleanObject(StringUtils.isNotBlank(claim) ? claim : "false");
 	}
 
-	private static Map<String, List<Param>> parseParams(final Element e) {
+	private static List<Param> parseParams(final Element e) {
 		final List<Node> params = e.selectNodes("./param");
 		return params.stream()
 			.map(n -> (Element) n)
-			.map(p -> new Param()
-				.setName(p.attributeValue("name"))
-				.setValue(p.getTextTrim()))
-			.collect(Collectors.toMap(Param::getName, Lists::newArrayList, (p1, p2) -> {
-				final List<Param> p = new ArrayList<>(p1);
-				p.addAll(p2);
-				return p;
-			}));
+			.map(p -> new Param().setName(p.attributeValue("name")).setValue(p.getTextTrim()))
+			.collect(Collectors.toList());
 	}
 
 	public static FunderDetails asFunderDetails(final Context c) {

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/context/ContextNotFoundException.java

@@ -1,23 +0,0 @@
-package eu.dnetlib.openaire.context;
-
-import org.springframework.http.HttpStatus;
-import org.springframework.web.bind.annotation.ResponseBody;
-import org.springframework.web.bind.annotation.ResponseStatus;
-
-@ResponseBody
-@ResponseStatus(value = HttpStatus.NOT_FOUND)
-public class ContextNotFoundException extends Exception {
-
-	/**
-	 *
-	 */
-	private static final long serialVersionUID = -2026506752817353752L;
-
-	public ContextNotFoundException(final String msg) {
-		super(msg);
-	}
-
-	public ContextNotFoundException(final Exception e) {
-		super(e);
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/DsmApiController.java

@@ -24,25 +24,23 @@ import org.springframework.web.bind.annotation.RequestBody;
 import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.enabling.datasources.common.DsmForbiddenException;
-import eu.dnetlib.enabling.datasources.common.DsmNotFoundException;
 import eu.dnetlib.openaire.common.AbstractExporterController;
 import eu.dnetlib.openaire.common.OperationManager;
-import eu.dnetlib.openaire.dsm.domain.AggregationHistoryResponse;
-import eu.dnetlib.openaire.dsm.domain.ApiDetails;
-import eu.dnetlib.openaire.dsm.domain.ApiDetailsResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetails;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailsUpdate;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailsWithApis;
-import eu.dnetlib.openaire.dsm.domain.DatasourceSnippetResponse;
-import eu.dnetlib.openaire.dsm.domain.RegisteredDatasourceInfo;
-import eu.dnetlib.openaire.dsm.domain.RequestFilter;
-import eu.dnetlib.openaire.dsm.domain.RequestSort;
-import eu.dnetlib.openaire.dsm.domain.RequestSortOrder;
-import eu.dnetlib.openaire.dsm.domain.SimpleResponse;
-import eu.dnetlib.openaire.vocabularies.Country;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationHistoryResponseV1;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetailsResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailsUpdate;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailsWithApis;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceSnippetResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.RegisteredDatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestFilter;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSort;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSortOrder;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleResponse;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Country;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.responses.ApiResponses;
@@ -69,7 +67,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public List<Country> listCountries() throws DsmException {
+	public List<Country> listCountries() throws DsmApiException {
 		return dsmCore.listCountries();
 	}
 
@@ -88,7 +86,7 @@ public class DsmApiController extends AbstractExporterController {
 		@RequestParam final RequestSortOrder order,
 		@RequestBody final RequestFilter requestFilter,
 		@PathVariable final int page,
-		@PathVariable final int size) throws DsmException {
+		@PathVariable final int size) throws DsmApiException {
 		final StopWatch stop = StopWatch.createStarted();
 		final DatasourceDetailResponse rsp = dsmCore.searchDsDetails(requestSortBy, order, requestFilter, page, size);
 		return prepareResponse(page, size, stop, rsp);
@@ -97,16 +95,16 @@ public class DsmApiController extends AbstractExporterController {
 	@GetMapping(value = "/ds/aggregationhistory/{dsId}", produces = {
 		"application/json"
 	})
-	@Operation(summary = "search datasources", description = "Returns list of Datasource details.", tags = {
+	@Operation(summary = "return aggregation hitory", description = "Returns the aggregation history of a datasource", tags = {
 		DS, R
 	})
 	@ApiResponses(value = {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public AggregationHistoryResponse aggregationHistory(@PathVariable final String dsId) throws DsmException {
+	public AggregationHistoryResponseV1 aggregationHistory(@PathVariable final String dsId) throws DsmApiException {
 		final StopWatch stop = StopWatch.createStarted();
-		final AggregationHistoryResponse rsp = dsmCore.aggregationhistory(dsId);
+		final AggregationHistoryResponseV1 rsp = dsmCore.aggregationhistoryV1(dsId);
 		return prepareResponse(0, rsp.getAggregationInfo().size(), stop, rsp);
 	}
 
@@ -125,7 +123,7 @@ public class DsmApiController extends AbstractExporterController {
 		@RequestParam final RequestSortOrder order,
 		@RequestBody final RequestFilter requestFilter,
 		@PathVariable final int page,
-		@PathVariable final int size) throws DsmException {
+		@PathVariable final int size) throws DsmApiException {
 		final StopWatch stop = StopWatch.createStarted();
 		final DatasourceSnippetResponse rsp = dsmCore.searchSnippet(requestSortBy, order, requestFilter, page, size);
 		return prepareResponse(page, size, stop, rsp);
@@ -147,7 +145,7 @@ public class DsmApiController extends AbstractExporterController {
 		@RequestParam final RequestSortOrder order,
 		@RequestBody final RequestFilter requestFilter,
 		@PathVariable final int page,
-		@PathVariable final int size) throws DsmException {
+		@PathVariable final int size) throws DsmApiException {
 		final StopWatch stop = StopWatch.createStarted();
 		final DatasourceSnippetResponse rsp = dsmCore.searchRegistered(requestSortBy, order, requestFilter, page, size);
 		return prepareResponse(page, size, stop, rsp);
@@ -198,7 +196,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
 	public ApiDetailsResponse getApi(
-		@PathVariable final String dsId) throws DsmException {
+		@PathVariable final String dsId) throws DsmApiException {
 
 		final StopWatch stop = StopWatch.createStarted();
 		final ApiDetailsResponse rsp = dsmCore.getApis(dsId);
@@ -218,7 +216,7 @@ public class DsmApiController extends AbstractExporterController {
 	public List<String> searchBaseUrls(
 		@RequestBody final RequestFilter requestFilter,
 		@PathVariable final int page,
-		@PathVariable final int size) throws DsmException {
+		@PathVariable final int size) throws DsmApiException {
 
 		return dsmCore.findBaseURLs(requestFilter, page, size);
 	}
@@ -233,7 +231,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "403", description = "Api not removable"),
 		@ApiResponse(responseCode = "500", description = "DSM Server error")
 	})
-	public void deleteApi(@PathVariable final String apiId) throws DsmForbiddenException, DsmNotFoundException {
+	public void deleteApi(@PathVariable final String apiId) throws DsmApiException {
 		dsmCore.deleteApi(apiId);
 	}
 
@@ -247,7 +245,7 @@ public class DsmApiController extends AbstractExporterController {
 	})
 	public void setManaged(
 		@RequestParam final String id,
-		@RequestParam final boolean managed) throws DsmException {
+		@RequestParam final boolean managed) throws DsmApiException {
 
 		dsmCore.setManaged(id, managed);
 	}
@@ -260,7 +258,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public boolean isManaged(@PathVariable final String id) throws DsmException {
+	public boolean isManaged(@PathVariable final String id) throws DsmApiException {
 		return dsmCore.isManaged(id);
 	}
 
@@ -273,10 +271,10 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "400", description = "Malformed request"),
 		@ApiResponse(responseCode = "500", description = "Unexpected error")
 	})
-	public void saveDs(@Valid @RequestBody final DatasourceDetails datasource) throws DsmException {
+	public void saveDs(@Valid @RequestBody final DatasourceDetails datasource) throws DsmApiException {
 
 		if (dsmCore.exist(datasource)) { // TODO further check that the DS doesn't have any API
-			throw new DsmException(HttpStatus.SC_CONFLICT, String.format("cannot register, datasource already defined '%s'", datasource.getId()));
+			throw new DsmApiException(HttpStatus.SC_CONFLICT, String.format("cannot register, datasource already defined '%s'", datasource.getId()));
 		}
 		dsmCore.save(datasource);
 	}
@@ -290,10 +288,10 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "400", description = "Malformed request"),
 		@ApiResponse(responseCode = "500", description = "Unexpected error")
 	})
-	public void saveDsWithApis(@Valid @RequestBody final DatasourceDetailsWithApis d) throws DsmException {
-		if (d.getDatasource() == null) { throw new DsmException(HttpStatus.SC_BAD_REQUEST, "Datasource field is null"); }
+	public void saveDsWithApis(@Valid @RequestBody final DatasourceDetailsWithApis d) throws DsmApiException {
+		if (d.getDatasource() == null) { throw new DsmApiException(HttpStatus.SC_BAD_REQUEST, "Datasource field is null"); }
 		if (dsmCore.exist(d.getDatasource())) { // TODO further check that the DS doesn't have any API
-			throw new DsmException(HttpStatus.SC_CONFLICT, String.format("cannot register, datasource already defined '%s'", d.getDatasource().getId()));
+			throw new DsmApiException(HttpStatus.SC_CONFLICT, String.format("cannot register, datasource already defined '%s'", d.getDatasource().getId()));
 		}
 		dsmCore.save(d);
 	}
@@ -307,7 +305,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
 	public void updateDatasource(
-		@RequestBody final DatasourceDetailsUpdate ds) throws DsmException, DsmNotFoundException {
+		@RequestBody final DatasourceDetailsUpdate ds) throws DsmApiException {
 
 		dsmCore.updateDatasource(ds);
 	}
@@ -323,7 +321,7 @@ public class DsmApiController extends AbstractExporterController {
 	public void updateBaseUrl(
 		@RequestParam final String dsId,
 		@RequestParam final String apiId,
-		@RequestParam final String baseUrl) throws DsmException {
+		@RequestParam final String baseUrl) throws DsmApiException {
 
 		dsmCore.updateApiBaseurl(dsId, apiId, baseUrl);
 	}
@@ -340,7 +338,7 @@ public class DsmApiController extends AbstractExporterController {
 		@RequestParam final String dsId,
 		@RequestParam final String apiId,
 		@RequestParam final String compliance,
-		@RequestParam(required = false, defaultValue = "false") final boolean override) throws DsmException {
+		@RequestParam(required = false, defaultValue = "false") final boolean override) throws DsmApiException {
 
 		dsmCore.updateApiCompatibility(dsId, apiId, compliance, override);
 	}
@@ -356,7 +354,7 @@ public class DsmApiController extends AbstractExporterController {
 	public void updateOaiSetl(
 		@RequestParam final String dsId,
 		@RequestParam final String apiId,
-		@RequestParam final String oaiSet) throws DsmException {
+		@RequestParam final String oaiSet) throws DsmApiException {
 
 		dsmCore.updateApiOaiSet(dsId, apiId, oaiSet);
 	}
@@ -369,8 +367,8 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public void addApi(@RequestBody final ApiDetails api) throws DsmException {
-		if (StringUtils.isBlank(api.getDatasource())) { throw new DsmException(HttpStatus.SC_BAD_REQUEST, "missing datasource id"); }
+	public void addApi(@RequestBody final ApiDetails api) throws DsmApiException {
+		if (StringUtils.isBlank(api.getDatasource())) { throw new DsmApiException(HttpStatus.SC_BAD_REQUEST, "missing datasource id"); }
 		dsmCore.addApi(api);
 	}
 
@@ -387,7 +385,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public int getOps() throws DsmException {
+	public int getOps() throws DsmApiException {
 		return operationManager.getOpSize();
 	}
 
@@ -399,7 +397,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public int killOps() throws DsmException {
+	public int killOps() throws DsmApiException {
 		return operationManager.dropAll();
 	}
 
@@ -411,7 +409,7 @@ public class DsmApiController extends AbstractExporterController {
 		@ApiResponse(responseCode = "200", description = "OK"),
 		@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public void dropCache() throws DsmException {
+	public void dropCache() throws DsmApiException {
 		dsmCore.dropCaches();
 	}
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/DsmApiControllerV2.java

@@ -17,8 +17,10 @@ import org.springframework.web.bind.annotation.RestController;
 
 import eu.dnetlib.openaire.common.AbstractExporterController;
 import eu.dnetlib.openaire.dsm.dao.ResponseUtils;
-import eu.dnetlib.openaire.dsm.domain.SimpleDatasourceInfo;
-import eu.dnetlib.openaire.dsm.domain.SimpleResponse;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationHistoryResponseV2;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleDatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleResponse;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.responses.ApiResponses;
@@ -83,4 +85,20 @@ public class DsmApiControllerV2 extends AbstractExporterController {
 		return prepareResponse(1, list.size(), stop, rsp);
 	}
 
+	@GetMapping(value = "/aggregationhistory/{dsId}", produces = {
+		"application/json"
+	})
+	@Operation(summary = "Return history", description = "Returns the aggregation history of the datasources (it includes also the failures)", tags = {
+		DS, R
+	})
+	@ApiResponses(value = {
+		@ApiResponse(responseCode = "200", description = "OK"),
+		@ApiResponse(responseCode = "500", description = "unexpected error")
+	})
+	public AggregationHistoryResponseV2 aggregationHistory(@PathVariable final String dsId) throws DsmApiException {
+		final StopWatch stop = StopWatch.createStarted();
+		final AggregationHistoryResponseV2 rsp = dsmCore.aggregationhistoryV2(dsId);
+		return prepareResponse(0, rsp.getAggregationInfo().size(), stop, rsp);
+	}
+
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/DsmCore.java

@@ -9,6 +9,7 @@ import java.nio.charset.Charset;
 import java.util.Collection;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Objects;
 import java.util.Set;
 import java.util.function.Function;
 import java.util.stream.Collectors;
@@ -29,35 +30,41 @@ import org.springframework.jdbc.core.JdbcTemplate;
 import org.springframework.jdbc.core.RowMapper;
 import org.springframework.stereotype.Component;
 
-import eu.dnetlib.enabling.datasources.common.AggregationInfo;
+import com.google.common.collect.Lists;
+
 import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.enabling.datasources.common.DsmForbiddenException;
 import eu.dnetlib.enabling.datasources.common.DsmNotFoundException;
 import eu.dnetlib.openaire.common.ISClient;
-import eu.dnetlib.openaire.community.CommunityClient;
+import eu.dnetlib.openaire.dsm.dao.CountryTermRepository;
 import eu.dnetlib.openaire.dsm.dao.DatasourceDao;
 import eu.dnetlib.openaire.dsm.dao.MongoLoggerClient;
 import eu.dnetlib.openaire.dsm.dao.ResponseUtils;
 import eu.dnetlib.openaire.dsm.dao.VocabularyClient;
 import eu.dnetlib.openaire.dsm.dao.utils.DsmMappingUtils;
-import eu.dnetlib.openaire.dsm.domain.AggregationHistoryResponse;
-import eu.dnetlib.openaire.dsm.domain.ApiDetails;
-import eu.dnetlib.openaire.dsm.domain.ApiDetailsResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetails;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailsUpdate;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailsWithApis;
-import eu.dnetlib.openaire.dsm.domain.DatasourceSnippetResponse;
-import eu.dnetlib.openaire.dsm.domain.RegisteredDatasourceInfo;
-import eu.dnetlib.openaire.dsm.domain.RequestFilter;
-import eu.dnetlib.openaire.dsm.domain.RequestSort;
-import eu.dnetlib.openaire.dsm.domain.RequestSortOrder;
-import eu.dnetlib.openaire.dsm.domain.SimpleDatasourceInfo;
-import eu.dnetlib.openaire.dsm.domain.SimpleResponse;
 import eu.dnetlib.openaire.dsm.domain.db.ApiDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.IdentityDbEntry;
-import eu.dnetlib.openaire.vocabularies.Country;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationHistoryResponseV1;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationHistoryResponseV2;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfoV1;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetailsResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailsUpdate;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailsWithApis;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceSnippetResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.RegisteredDatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestFilter;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSort;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSortOrder;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleDatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleResponse;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Country;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Vocabulary;
 
 @Component
 @ConditionalOnProperty(value = "openaire.exporter.enable.dsm", havingValue = "true")
@@ -78,18 +85,20 @@ public class DsmCore {
 	private DatasourceDao<DatasourceDbEntry, ApiDbEntry> dsDao;
 
 	@Autowired
-	private JdbcTemplate jdbcTemplate;
+	private CountryTermRepository countryTermRepository;
 
 	@Autowired
-	private CommunityClient communityClient;
+	private JdbcTemplate jdbcTemplate;
 
-	public List<Country> listCountries() throws DsmException {
-		try {
-			return dsDao.listCountries();
-		} catch (final Throwable e) {
-			log.error("error listing countries", e);
-			throw e;
-		}
+	public List<Country> listCountries() throws DsmApiException {
+		final List<Country> countries = Lists.newArrayList();
+		final Vocabulary v = vocabularyClient.getCountries();
+		countries.addAll(countryTermRepository.findAll()
+			.stream()
+			.filter(Objects::nonNull)
+			.map(t -> new Country(t.getTerm(), v.getEnglishName(t.getTerm())))
+			.collect(Collectors.toList()));
+		return countries;
 	}
 
 	public DatasourceDetailResponse searchDsDetails(final RequestSort requestSortBy,
@@ -97,14 +106,14 @@ public class DsmCore {
 		final RequestFilter requestFilter,
 		final int page,
 		final int size)
-		throws DsmException {
+		throws DsmApiException {
 
 		try {
 			final Page<DatasourceDbEntry> dsPage = dsDao.search(requestSortBy, order, requestFilter, page, size);
 			return ResponseUtils.detailsResponse(dsPage.map(d -> asDetails(d)).getContent(), dsPage.getTotalElements());
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error("error searching datasources", e);
-			throw e;
+			throw new DsmApiException(e.getCode(), "error searching datasources", e);
 		}
 	}
 
@@ -113,13 +122,13 @@ public class DsmCore {
 		final RequestFilter requestFilter,
 		final int page,
 		final int size)
-		throws DsmException {
+		throws DsmApiException {
 		try {
 			final Page<DatasourceDbEntry> dsPage = dsDao.search(requestSortBy, order, requestFilter, page, size);
 			return ResponseUtils.snippetResponse(dsPage.map(DsmMappingUtils::asSnippetExtended).getContent(), dsPage.getTotalElements());
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error("error searching datasources", e);
-			throw e;
+			throw new DsmApiException(e.getCode(), "error searching datasources", e);
 		}
 	}
 
@@ -128,26 +137,26 @@ public class DsmCore {
 		final RequestFilter requestFilter,
 		final int page,
 		final int size)
-		throws DsmException {
+		throws DsmApiException {
 		try {
 			final Page<DatasourceDbEntry> dsPage = dsDao.searchRegistered(requestSortBy, order, requestFilter, page, size);
 			return ResponseUtils.snippetResponse(dsPage.map(DsmMappingUtils::asSnippetExtended).getContent(), dsPage.getTotalElements());
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error("error searching datasources", e);
-			throw e;
+			throw new DsmApiException(e.getCode(), "error searching datasources", e);
 		}
 	}
 
-	public List<String> findBaseURLs(final RequestFilter requestFilter, final int page, final int size) throws DsmException {
+	public List<String> findBaseURLs(final RequestFilter requestFilter, final int page, final int size) throws DsmApiException {
 		try {
 			return dsDao.findApiBaseURLs(requestFilter, page, size);
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error("error searching datasource base urls", e);
-			throw e;
+			throw new DsmApiException(e.getCode(), "error searching datasource base urls", e);
 		}
 	}
 
-	public ApiDetailsResponse getApis(final String dsId) throws DsmException {
+	public ApiDetailsResponse getApis(final String dsId) throws DsmApiException {
 		try {
 			final DatasourceDbEntry ds = dsDao.getDs(dsId);
 			final List<? extends ApiDbEntry> apis = dsDao.getApis(dsId);
@@ -157,37 +166,49 @@ public class DsmCore {
 				.map(a -> a.setTypology(ds.getTypology()))
 				.collect(Collectors.toList());
 			return ResponseUtils.apiResponse(api, api.size());
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error(String.format("error searching datasource api %s", dsId), e);
-			throw e;
+			throw new DsmApiException(e.getCode(), String.format("error searching datasource api %s", dsId), e);
 		}
 	}
 
-	public void setManaged(final String dsId, final boolean managed) throws DsmException {
+	public void setManaged(final String dsId, final boolean managed) throws DsmApiException {
 		log.info(String.format("updated ds '%s' managed with '%s'", dsId, managed));
-		dsDao.setManaged(dsId, managed);
+		try {
+			dsDao.setManaged(dsId, managed);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public boolean isManaged(final String dsId) throws DsmException {
-		return dsDao.isManaged(dsId);
+	public boolean isManaged(final String dsId) throws DsmApiException {
+		try {
+			return dsDao.isManaged(dsId);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public boolean exist(final DatasourceDetails d) throws DsmException {
-		return dsDao.existDs(d.getId());
+	public boolean exist(final DatasourceDetails d) throws DsmApiException {
+		try {
+			return dsDao.existDs(d.getId());
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public void save(final DatasourceDetails d) throws DsmException {
+	public void save(final DatasourceDetails d) throws DsmApiException {
 		try {
 			dsDao.saveDs(asDbEntry(d));
 			log.info("DS saved, " + d.getId());
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error(ExceptionUtils.getStackTrace(e));
-			throw e;
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
 		}
 	}
 
 	@Transactional
-	public void save(final DatasourceDetailsWithApis d) throws DsmException {
+	public void save(final DatasourceDetailsWithApis d) throws DsmApiException {
 		try {
 			dsDao.saveDs(asDbEntry(d.getDatasource()));
 			final List<ApiDetails> apis = d.getApis();
@@ -197,15 +218,13 @@ public class DsmCore {
 					addApi(api);
 				}
 			}
-		} catch (
-
-		final Throwable e) {
+		} catch (final DsmException e) {
 			log.error(ExceptionUtils.getStackTrace(e));
-			throw e;
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
 		}
 	}
 
-	public void updateDatasource(final DatasourceDetailsUpdate d) throws DsmException, DsmNotFoundException {
+	public void updateDatasource(final DatasourceDetailsUpdate d) throws DsmApiException {
 		try {
 			// initialize with current values from DB
 			final DatasourceDbEntry ds = dsDao.getDs(d.getId());
@@ -226,53 +245,78 @@ public class DsmCore {
 			}
 
 			dsDao.saveDs(ds);
-		} catch (final Throwable e) {
+		} catch (final DsmException e) {
 			log.error(ExceptionUtils.getStackTrace(e));
-			throw e;
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
 		}
 	}
 
 	// TODO remove if unused
-	public void deleteDs(final String dsId) throws DsmException {
+	public void deleteDs(final String dsId) throws DsmApiException {
 		log.info(String.format("deleted datasource '%s'", dsId));
-		dsDao.deleteDs(dsId);
+		try {
+			dsDao.deleteDs(dsId);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
 	// API
 
-	public void updateApiOaiSet(final String dsId, final String apiId, final String oaiSet) throws DsmException {
-		dsDao.upsertApiOaiSet(apiId, oaiSet);
+	public void updateApiOaiSet(final String dsId, final String apiId, final String oaiSet) throws DsmApiException {
+		try {
+			dsDao.upsertApiOaiSet(apiId, oaiSet);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public void updateApiBaseurl(final String dsId, final String apiId, final String baseUrl) throws DsmException {
+	public void updateApiBaseurl(final String dsId, final String apiId, final String baseUrl) throws DsmApiException {
 		log.info(String.format("updated api '%s' baseurl with '%s'", apiId, baseUrl));
-		dsDao.updateApiBaseUrl(apiId, baseUrl);
+		try {
+			dsDao.updateApiBaseUrl(apiId, baseUrl);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public void updateApiCompatibility(final String dsId, final String apiId, final String compliance, final boolean override) throws DsmException {
+	public void updateApiCompatibility(final String dsId, final String apiId, final String compliance, final boolean override) throws DsmApiException {
 		log.info(String.format("updated api '%s' compliance with '%s'", apiId, compliance));
-		dsDao.updateCompliance(null, apiId, compliance, override);
+		try {
+			dsDao.updateCompliance(null, apiId, compliance, override);
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public void addApi(final ApiDetails api) throws DsmException {
+	public void addApi(final ApiDetails api) throws DsmApiException {
 		if (StringUtils.isBlank(api.getId())) {
 			api.setId(createId(api));
 			log.info(String.format("missing api id, created '%s'", api.getId()));
 		}
-		dsDao.addApi(asDbEntry(api));
-		log.info("API saved, id: " + api.getId());
+		try {
+			dsDao.addApi(asDbEntry(api));
+			log.info("API saved, id: " + api.getId());
+		} catch (final DsmException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
-	public void deleteApi(final String apiId) throws DsmForbiddenException, DsmNotFoundException {
+	public void deleteApi(final String apiId) throws DsmApiException {
 		// TODO handle the api removal in case of associated workflows.
-		dsDao.deleteApi(null, apiId);
+		try {
+			dsDao.deleteApi(null, apiId);
+		} catch (final DsmForbiddenException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		} catch (final DsmNotFoundException e) {
+			throw new DsmApiException(e.getCode(), e.getMessage(), e);
+		}
 	}
 
 	public void dropCaches() {
 		mongoLoggerClient.dropCache();
 		isClient.dropCache();
 		vocabularyClient.dropCache();
-		communityClient.dropCache();
 	}
 
 	// HELPERS //////////////
@@ -313,9 +357,17 @@ public class DsmCore {
 		}
 	}
 
-	public AggregationHistoryResponse aggregationhistory(final String dsId) throws DsmException {
-		final List<AggregationInfo> history = mongoLoggerClient.getAggregationHistory(dsId);
-		final AggregationHistoryResponse rsp = new AggregationHistoryResponse(history);
+	@Deprecated
+	public AggregationHistoryResponseV1 aggregationhistoryV1(final String dsId) throws DsmApiException {
+		final List<AggregationInfoV1> history = mongoLoggerClient.getAggregationHistoryV1(dsId);
+		final AggregationHistoryResponseV1 rsp = new AggregationHistoryResponseV1(history);
+		rsp.setHeader(ResponseUtils.header(history.size()));
+		return rsp;
+	}
+
+	public AggregationHistoryResponseV2 aggregationhistoryV2(final String dsId) throws DsmApiException {
+		final List<AggregationInfo> history = mongoLoggerClient.getAggregationHistoryV2(dsId);
+		final AggregationHistoryResponseV2 rsp = new AggregationHistoryResponseV2(history);
 		rsp.setHeader(ResponseUtils.header(history.size()));
 		return rsp;
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/DatasourceDao.java

@@ -8,17 +8,14 @@ import eu.dnetlib.enabling.datasources.common.Api;
 import eu.dnetlib.enabling.datasources.common.Datasource;
 import eu.dnetlib.enabling.datasources.common.DatasourceManagerCommon;
 import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.openaire.dsm.domain.RequestFilter;
-import eu.dnetlib.openaire.dsm.domain.RequestSort;
-import eu.dnetlib.openaire.dsm.domain.RequestSortOrder;
-import eu.dnetlib.openaire.vocabularies.Country;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestFilter;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSort;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSortOrder;
 
 public interface DatasourceDao<DS extends Datasource<?, ?, ?>, API extends Api<?>> extends DatasourceManagerCommon<DS, API> {
 
 	// DATASOURCE
 
-	List<Country> listCountries() throws DsmException;
-
 	boolean existDs(final String dsId) throws DsmException;
 
 	Page<DS> search(RequestSort requestSortBy, RequestSortOrder order, RequestFilter requestFilter, int page, int size) throws DsmException;
@@ -55,6 +52,4 @@ public interface DatasourceDao<DS extends Datasource<?, ?, ?>, API extends Api<?
 
 	void updateApiBaseUrl(String apiId, String baseUrl) throws DsmException;
 
-	@Override
-	void addApi(final API api) throws DsmException;
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/DatasourceDaoImpl.java

@@ -9,7 +9,6 @@ import static eu.dnetlib.openaire.dsm.dao.DatasourceSpecs.dsSpec;
 import java.sql.Date;
 import java.util.HashSet;
 import java.util.List;
-import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
 
@@ -33,15 +32,13 @@ import eu.dnetlib.DnetOpenaireExporterProperties;
 import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.enabling.datasources.common.DsmForbiddenException;
 import eu.dnetlib.enabling.datasources.common.DsmNotFoundException;
-import eu.dnetlib.openaire.dsm.domain.RequestFilter;
-import eu.dnetlib.openaire.dsm.domain.RequestSort;
-import eu.dnetlib.openaire.dsm.domain.RequestSortOrder;
 import eu.dnetlib.openaire.dsm.domain.db.ApiDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.ApiParamDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceApiDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceDbEntry;
-import eu.dnetlib.openaire.vocabularies.Country;
-import eu.dnetlib.openaire.vocabularies.Vocabulary;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestFilter;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSort;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSortOrder;
 
 /**
  * Created by claudio on 20/10/2016.
@@ -55,9 +52,6 @@ public class DatasourceDaoImpl implements DatasourceDao<DatasourceDbEntry, ApiDb
 	@Autowired
 	private DnetOpenaireExporterProperties config;
 
-	@Autowired
-	private CountryTermRepository countryTermRepository;
-
 	@Autowired
 	private DatasourceDbEntryRepository dsRepository;
 
@@ -67,20 +61,8 @@ public class DatasourceDaoImpl implements DatasourceDao<DatasourceDbEntry, ApiDb
 	@Autowired
 	private DatasourceApiDbEntryRepository dsApiRepository;
 
-	@Autowired
-	private VocabularyClient vocabularyClient;
-
-	@Override
-	public List<Country> listCountries() throws DsmException {
-		final List<Country> countries = Lists.newArrayList();
-		final Vocabulary v = vocabularyClient.getCountries();
-		countries.addAll(countryTermRepository.findAll()
-			.stream()
-			.filter(Objects::nonNull)
-			.map(t -> new Country(t.getTerm(), v.getEnglishName(t.getTerm())))
-			.collect(Collectors.toList()));
-		return countries;
-	}
+	// @Autowired
+	// private VocabularyClient vocabularyClient;
 
 	@Override
 	public Page<DatasourceDbEntry> search(final RequestSort requestSortBy,
@@ -239,13 +221,11 @@ public class DatasourceDaoImpl implements DatasourceDao<DatasourceDbEntry, ApiDb
 
 	@Override
 	public void updateEoscDatasourceType(final String dsId, final String type) throws DsmException {
-		final Vocabulary typologies = vocabularyClient.getDatasourceTypologies();
-		if (!typologies.hasCode(type)) {
-			throw new DsmException(
-				HttpStatus.SC_BAD_REQUEST,
-				String.format("invalid datasource type '%s', provide one according to vocabulary %s", type, config.getVocabularies()
-					.getDatasourceTypologiesEndpoint()));
-		}
+		/*
+		 * final Vocabulary typologies = vocabularyClient.getDatasourceTypologies(); if (!typologies.hasCode(type)) { throw new
+		 * DsmException( HttpStatus.SC_BAD_REQUEST, String.format("invalid datasource type '%s', provide one according to vocabulary %s",
+		 * type, config.getVocabularies() .getDatasourceTypologiesEndpoint())); }
+		 */
 		dsRepository.setEoscDatasourceType(dsId, type);
 	}
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/DatasourceIndexClient.java

@@ -2,16 +2,14 @@ package eu.dnetlib.openaire.dsm.dao;
 
 import java.util.Queue;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.openaire.dsm.dao.utils.IndexDsInfo;
 import eu.dnetlib.openaire.dsm.dao.utils.IndexRecordsInfo;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
 
 public interface DatasourceIndexClient {
 
-	IndexRecordsInfo getIndexInfo(final String dsId, final IndexDsInfo info, final Queue<Throwable> errors) throws DsmException;
-
-	String getLastIndexingDate(final IndexDsInfo info) throws DsmException;
-
+	IndexRecordsInfo getIndexInfo(final String dsId, final IndexDsInfo info, final Queue<Throwable> errors) throws DsmApiException;
 
+	String getLastIndexingDate(final IndexDsInfo info) throws DsmApiException;
 
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/DatasourceIndexClientImpl.java

@@ -5,19 +5,15 @@ import java.util.Date;
 import java.util.List;
 import java.util.Map;
 import java.util.Queue;
-import java.util.concurrent.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
 import javax.annotation.PostConstruct;
 import javax.annotation.PreDestroy;
 
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import com.google.common.util.concurrent.*;
-import eu.dnetlib.DnetOpenaireExporterProperties;
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.miscutils.functional.hash.Hashing;
-import eu.dnetlib.openaire.dsm.dao.utils.DsmMappingUtils;
-import eu.dnetlib.openaire.dsm.dao.utils.IndexDsInfo;
-import eu.dnetlib.openaire.dsm.dao.utils.IndexRecordsInfo;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.time.DateFormatUtils;
 import org.apache.commons.logging.Log;
@@ -33,6 +29,22 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
 import org.springframework.http.HttpStatus;
 import org.springframework.stereotype.Component;
 
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListeningExecutorService;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import eu.dnetlib.DnetOpenaireExporterProperties;
+import eu.dnetlib.enabling.datasources.common.DsmException;
+import eu.dnetlib.miscutils.functional.hash.Hashing;
+import eu.dnetlib.openaire.dsm.dao.utils.DsmMappingUtils;
+import eu.dnetlib.openaire.dsm.dao.utils.IndexDsInfo;
+import eu.dnetlib.openaire.dsm.dao.utils.IndexRecordsInfo;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+
 /**
  * Created by claudio on 20/10/2016.
  */
@@ -54,9 +66,8 @@ public class DatasourceIndexClientImpl implements DatasourceIndexClient {
 
 	@PostConstruct
 	public void init() {
-		executor = MoreExecutors.listeningDecorator(
-				new ScheduledThreadPoolExecutor(5,
-						new ThreadFactoryBuilder().setNameFormat("datasource-index-client-%d").build()));
+		executor = MoreExecutors.listeningDecorator(new ScheduledThreadPoolExecutor(5,
+			new ThreadFactoryBuilder().setNameFormat("datasource-index-client-%d").build()));
 	}
 
 	@PreDestroy
@@ -64,65 +75,61 @@ public class DatasourceIndexClientImpl implements DatasourceIndexClient {
 		indexClientMap.forEach((name, client) -> {
 			try {
 				client.close();
-			} catch (IOException e) {
+			} catch (final IOException e) {
 				log.warn(String.format("unable to gracefully shutdown client for index %s", name));
 			}
 		});
 	}
 
 	@Override
-	public IndexRecordsInfo getIndexInfo(final String dsId, final IndexDsInfo info, final Queue<Throwable> errors) throws DsmException {
+	public IndexRecordsInfo getIndexInfo(final String dsId, final IndexDsInfo info, final Queue<Throwable> errors) throws DsmApiException {
 		try {
 			final String collectedFrom = StringUtils.substringBefore(dsId, SEPARATOR) + SEPARATOR + Hashing.md5(StringUtils.substringAfter(dsId, SEPARATOR));
 			final CloudSolrClient indexClient = getIndexClient(info);
 			final CountDownLatch latch = new CountDownLatch(2);
 			final IndexRecordsInfo indexRecordInfo = new IndexRecordsInfo();
 
-			Futures.addCallback(
-					executor.submit(() -> setDateAndTotal(dsId, collectedFrom, indexClient)),
-					new FutureCallback<IndexRecordsInfo>() {
+			Futures.addCallback(executor.submit(() -> setDateAndTotal(dsId, collectedFrom, indexClient)), new FutureCallback<IndexRecordsInfo>() {
 
-						@Override
-						public void onSuccess(final IndexRecordsInfo info) {
-							indexRecordInfo
-									.setTotal(info.getTotal())
-									.setDate(info.getDate());
-							latch.countDown();
-						}
+				@Override
+				public void onSuccess(final IndexRecordsInfo info) {
+					indexRecordInfo
+						.setTotal(info.getTotal())
+						.setDate(info.getDate());
+					latch.countDown();
+				}
 
-						@Override
-						public void onFailure(final Throwable e) {
-							errors.offer(e);
-							latch.countDown();
-						}
-					}, executor);
+				@Override
+				public void onFailure(final Throwable e) {
+					errors.offer(e);
+					latch.countDown();
+				}
+			}, executor);
 
-			Futures.addCallback(
-					executor.submit(() -> setFunded(dsId, collectedFrom, indexClient)),
-					new FutureCallback<Long>() {
+			Futures.addCallback(executor.submit(() -> setFunded(dsId, collectedFrom, indexClient)), new FutureCallback<Long>() {
 
-						@Override
-						public void onSuccess(final Long numFound) {
-							indexRecordInfo.setFunded(numFound);
-							latch.countDown();
-						}
+				@Override
+				public void onSuccess(final Long numFound) {
+					indexRecordInfo.setFunded(numFound);
+					latch.countDown();
+				}
 
-						@Override
-						public void onFailure(final Throwable e) {
-							errors.offer(e);
-							latch.countDown();
-						}
-					}, executor);
+				@Override
+				public void onFailure(final Throwable e) {
+					errors.offer(e);
+					latch.countDown();
+				}
+			}, executor);
 
 			waitLatch(latch, errors, config.getRequestTimeout());
 			return indexRecordInfo;
 		} catch (final Throwable e) {
-			throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(), String.format("error reading index info", dsId), e);
+			throw new DsmApiException(HttpStatus.INTERNAL_SERVER_ERROR.value(), String.format("error reading index info", dsId), e);
 		}
 	}
 
 	@Override
-	public String getLastIndexingDate(final IndexDsInfo info) throws DsmException {
+	public String getLastIndexingDate(final IndexDsInfo info) throws DsmApiException {
 		try {
 			final SolrQuery query = new SolrQuery("oaftype:datasource").setRows(1);
 			final QueryResponse rsp = getIndexClient(info).query(query);
@@ -130,28 +137,28 @@ public class DatasourceIndexClientImpl implements DatasourceIndexClient {
 			final String dsversion = doc.get("__dsversion").toString();
 			return StringUtils.substringBefore(dsversion, "T");
 		} catch (SolrServerException | IOException e) {
-			throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Error querying index DS profile: " + info, e);
+			throw new DsmApiException(HttpStatus.INTERNAL_SERVER_ERROR.value(), "Error querying index DS profile: " + info, e);
 		}
 	}
 
 	private Long setFunded(
-			final String dsId,
-			final String collectedFrom,
-			final CloudSolrClient indexClient) throws DsmException {
+		final String dsId,
+		final String collectedFrom,
+		final CloudSolrClient indexClient) throws DsmException {
 		final String query =
-				String.format("oaftype:result AND deletedbyinference:false AND collectedfromdatasourceid:\"%s\" AND relprojectid:*", collectedFrom);
+			String.format("oaftype:result AND deletedbyinference:false AND collectedfromdatasourceid:\"%s\" AND relprojectid:*", collectedFrom);
 		log.debug(String.format("query: %s", query));
 		try {
 			return indexClient.query(new SolrQuery(query).setRows(0)).getResults().getNumFound();
-		} catch (Throwable e) {
+		} catch (final Throwable e) {
 			throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(), String.format("Error querying index for funded results '%s'", dsId), e);
 		}
 	}
 
 	private IndexRecordsInfo setDateAndTotal(
-			final String dsId,
-			final String collectedFrom,
-			final CloudSolrClient indexClient) throws DsmException {
+		final String dsId,
+		final String collectedFrom,
+		final CloudSolrClient indexClient) throws DsmException {
 		try {
 			final String query = String.format("oaftype:result AND deletedbyinference:false AND collectedfromdatasourceid:\"%s\"", collectedFrom);
 			log.debug(String.format("query: %s", query));
@@ -166,9 +173,9 @@ public class DatasourceIndexClientImpl implements DatasourceIndexClient {
 			// query: %s", queryTotal));
 			// }
 			return new IndexRecordsInfo()
-					.setDate(getDate(doc))
-					.setTotal(rsp.getResults().getNumFound());
-		} catch (Throwable e) {
+				.setDate(getDate(doc))
+				.setTotal(rsp.getResults().getNumFound());
+		} catch (final Throwable e) {
 			throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(), String.format("Error querying index for date and total '%s'", dsId), e);
 		}
 	}
@@ -176,8 +183,10 @@ public class DatasourceIndexClientImpl implements DatasourceIndexClient {
 	@SuppressWarnings("unchecked")
 	private String getDate(final SolrDocument doc) throws DsmException {
 		final List<Date> dsversion = (List<Date>) doc.get(DSVERSION);
-		if (dsversion == null || dsversion.isEmpty()) { throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(),
-				String.format("cannot find %s in matched solr document", DSVERSION)); }
+		if (dsversion == null || dsversion.isEmpty()) {
+			throw new DsmException(HttpStatus.INTERNAL_SERVER_ERROR.value(),
+				String.format("cannot find %s in matched solr document", DSVERSION));
+		}
 		final Date date = Iterables.getLast(dsversion);
 
 		return DateFormatUtils.format(date, DsmMappingUtils.DATE_FORMAT);

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/DatasourceSpecs.java

@@ -17,12 +17,12 @@ import org.apache.commons.logging.LogFactory;
 import org.springframework.data.jpa.domain.Specification;
 
 import eu.dnetlib.enabling.datasources.common.DsmRuntimeException;
-import eu.dnetlib.openaire.dsm.domain.FilterName;
-import eu.dnetlib.openaire.dsm.domain.RequestFilter;
-import eu.dnetlib.openaire.dsm.domain.RequestSort;
-import eu.dnetlib.openaire.dsm.domain.RequestSortOrder;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceApiDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceDbEntry;
+import eu.dnetlib.openaire.exporter.model.dsm.FilterName;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestFilter;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSort;
+import eu.dnetlib.openaire.exporter.model.dsm.RequestSortOrder;
 
 public class DatasourceSpecs {
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/MongoLoggerClient.java

@@ -1,13 +1,16 @@
 package eu.dnetlib.openaire.dsm.dao;
 
-import eu.dnetlib.enabling.datasources.common.AggregationInfo;
-import eu.dnetlib.enabling.datasources.common.DsmException;
-
 import java.util.List;
 
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfoV1;
+
 public interface MongoLoggerClient {
 
-	List<AggregationInfo> getAggregationHistory(final String dsId) throws DsmException;
+	List<AggregationInfoV1> getAggregationHistoryV1(final String dsId) throws DsmApiException;
+
+	List<AggregationInfo> getAggregationHistoryV2(final String dsId) throws DsmApiException;
 
 	void dropCache();
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/MongoLoggerClientImpl.java

@@ -41,15 +41,14 @@ import com.mongodb.client.MongoCollection;
 
 import eu.dnetlib.DnetOpenaireExporterProperties;
 import eu.dnetlib.DnetOpenaireExporterProperties.Datasource;
-import eu.dnetlib.enabling.datasources.common.AggregationInfo;
-import eu.dnetlib.enabling.datasources.common.AggregationStage;
-import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.miscutils.datetime.DateUtils;
 import eu.dnetlib.openaire.common.Utils;
 import eu.dnetlib.openaire.dsm.dao.utils.DsmMappingUtils;
-import eu.dnetlib.openaire.dsm.domain.CollectionInfo;
-import eu.dnetlib.openaire.dsm.domain.CollectionMode;
-import eu.dnetlib.openaire.dsm.domain.TransformationInfo;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfoV1;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationStage;
+import eu.dnetlib.openaire.exporter.model.dsm.CollectionMode;
 import eu.dnetlib.openaire.info.JdbcInfoDao;
 
 /**
@@ -71,39 +70,84 @@ public class MongoLoggerClientImpl implements MongoLoggerClient {
 	private JdbcInfoDao jdbcInfoDao;
 
 	private final static String LOADTIME = "loadtime";
-	private final LoadingCache<String, Instant> loadingCache = CacheBuilder.newBuilder()
-		.maximumSize(1)
-		.expireAfterWrite(60, TimeUnit.MINUTES)
-		.build(new CacheLoader<String, Instant>() {
+	private final LoadingCache<String, Instant> loadingCache =
+			CacheBuilder.newBuilder().maximumSize(1).expireAfterWrite(60, TimeUnit.MINUTES).build(new CacheLoader<String, Instant>() {
 
-			// The only cached value is associated to "loadtime"
-			@Override
-			public Instant load(final String key) {
-				final Instant loadTime = getLoadTime();
-				log.debug("found load time: " + loadTime.toString());
-				return loadTime;
-			}
-		});
+				// The only cached value is associated to "loadtime"
+				@Override
+				public Instant load(final String key) {
+					final Instant loadTime = getLoadTime();
+					log.debug("found load time: " + loadTime.toString());
+					return loadTime;
+				}
+			});
 
 	private static final Bson fields = getFields();
 
 	private static MongoCollection<Document> collection = null;
 
 	@Override
-	@Cacheable("dsm-aggregationhistory-cache")
-	public List<AggregationInfo> getAggregationHistory(final String dsId) throws DsmException {
+	@Cacheable("dsm-aggregationhistory-cache-v1")
+	@Deprecated
+	public List<AggregationInfoV1> getAggregationHistoryV1(final String dsId) throws DsmApiException {
+		return getAggregationHistoryV1(dsId, queryForAggregationHistoryV1(dsId, "(collect|transform)"), getMapperV1());
+	}
+
+	@Override
+	@Cacheable("dsm-aggregationhistory-cache-v2")
+	public List<AggregationInfo> getAggregationHistoryV2(final String dsId) throws DsmApiException {
+		return getAggregationHistoryV2(dsId, queryForAggregationHistoryV2(dsId, "(collect|transform)"), getMapperV2());
+	}
+
+	@Deprecated
+	private List<AggregationInfoV1> getAggregationHistoryV1(final String dsId,
+			final Bson queryForAggregationHistory,
+			final Function<Document, AggregationInfoV1> mapper) throws DsmApiException {
 		log.warn(String.format("getAggregationHistory(dsId = %s): not using cache", dsId));
 		final Datasource conf = config.getDatasource();
 		try {
-			final FindIterable<Document> aggregationDocs = getCollection().find(queryForAggregationHistory(dsId, "(collect|transform)"))
-				.projection(fields)
-				.limit(conf.getMongoQueryLimit())
-				.sort(dbo("system:startHumanDate", -1));
+			final FindIterable<Document> aggregationDocs = getCollection().find(queryForAggregationHistory)
+					.projection(fields)
+					.limit(conf.getMongoQueryLimit())
+					.sort(dbo("system:startHumanDate", -1));
+
+			final List<AggregationInfoV1> aggregationInfos = Utils.stream(aggregationDocs.iterator())
+					.map(mapper)
+					.filter(ai -> ai.getNumberOfRecords() >= 0 && StringUtils.isNotBlank(ai.getDate()))
+					.collect(Collectors.toList());
+
+			final Instant loadTime = loadingCache.get(LOADTIME);
+
+			if (!Objects.equals(Instant.MIN, loadTime)) {
+				for (final AggregationInfoV1 a : aggregationInfos) {
+					if (asInstant(a).isBefore(loadTime) && AggregationStage.COLLECT.equals(a.getAggregationStage())) {
+						a.setIndexedVersion(true);
+						break;
+					}
+				}
+			}
+
+			return aggregationInfos;
+		} catch (final Throwable e) {
+			throw new DsmApiException(HttpStatus.SC_INTERNAL_SERVER_ERROR, String.format("error reading aggregation history for '%s'", dsId), e);
+		}
+	}
+
+	private List<AggregationInfo> getAggregationHistoryV2(final String dsId,
+			final Bson queryForAggregationHistory,
+			final Function<Document, AggregationInfo> mapper) throws DsmApiException {
+		log.warn(String.format("getAggregationHistory(dsId = %s): not using cache", dsId));
+		final Datasource conf = config.getDatasource();
+		try {
+			final FindIterable<Document> aggregationDocs = getCollection().find(queryForAggregationHistory)
+					.projection(fields)
+					.limit(conf.getMongoQueryLimit())
+					.sort(dbo("system:startHumanDate", -1));
 
 			final List<AggregationInfo> aggregationInfos = Utils.stream(aggregationDocs.iterator())
-				.map(getMapper())
-				.filter(ai -> ai.getNumberOfRecords() >= 0 && StringUtils.isNotBlank(ai.getDate()))
-				.collect(Collectors.toList());
+					.map(mapper)
+					.filter(ai -> ai.getNumberOfRecords() >= 0 && StringUtils.isNotBlank(ai.getDate()))
+					.collect(Collectors.toList());
 
 			final Instant loadTime = loadingCache.get(LOADTIME);
 
@@ -118,7 +162,7 @@ public class MongoLoggerClientImpl implements MongoLoggerClient {
 
 			return aggregationInfos;
 		} catch (final Throwable e) {
-			throw new DsmException(HttpStatus.SC_INTERNAL_SERVER_ERROR, String.format("error reading aggregation history for '%s'", dsId), e);
+			throw new DsmApiException(HttpStatus.SC_INTERNAL_SERVER_ERROR, String.format("error reading aggregation history for '%s'", dsId), e);
 		}
 	}
 
@@ -132,69 +176,96 @@ public class MongoLoggerClientImpl implements MongoLoggerClient {
 		return Instant.parse(a.getDate() + "T00:00:00Z");
 	}
 
+	@Deprecated
+	private Instant asInstant(final AggregationInfoV1 a) {
+		return Instant.parse(a.getDate() + "T00:00:00Z");
+	}
+
 	@Override
 	@CacheEvict(cacheNames = {
-		"dsm-aggregationhistory-cache", "dsm-firstharvestdate-cache"
+			"dsm-aggregationhistory-cache-v1", "dsm-aggregationhistory-cache-v2", "dsm-firstharvestdate-cache"
 	}, allEntries = true)
 	@Scheduled(fixedDelayString = "${openaire.exporter.cache.ttl}")
 	public void dropCache() {
 		log.debug("dropped dsManager aggregation history cache");
 	}
 
-	private Function<Document, AggregationInfo> getMapper() {
-		return new Function<Document, AggregationInfo>() {
+	@Deprecated
+	private Function<Document, AggregationInfoV1> getMapperV1() {
+		return d -> {
 
-			@Override
-			public AggregationInfo apply(final Document d) {
+			AggregationInfoV1 info = null;
+			final AggregationStage stage = AggregationStage.parse(d.getString("system:wfName"));
+			final boolean success = isCompletedSuccesfully(d);
 
-				AggregationInfo info = null;
-				final AggregationStage stage = AggregationStage.parse(d.getString("system:wfName"));
-				final boolean success = isCompletedSuccesfully(d);
+			switch (stage) {
 
-				switch (stage) {
-
-				case COLLECT:
-					final CollectionInfo cInfo = new CollectionInfo();
-					cInfo.setAggregationStage(stage);
-					cInfo.setCollectionMode(getCollectionMode(d));
-					cInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
-					cInfo.setDate(getDate(d));
-					cInfo.setCompletedSuccessfully(success);
-					info = cInfo;
-					break;
-				case TRANSFORM:
-					final TransformationInfo tInfo = new TransformationInfo();
-					tInfo.setAggregationStage(stage);
-					tInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
-					tInfo.setDate(getDate(d));
-					tInfo.setCompletedSuccessfully(success);
-					info = tInfo;
-					break;
-				}
-				return info;
+			case COLLECT:
+				final AggregationInfoV1 cInfo = new AggregationInfoV1();
+				cInfo.setAggregationStage(stage);
+				cInfo.setCollectionMode(getCollectionMode(d));
+				cInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
+				cInfo.setDate(getDate(d));
+				info = cInfo;
+				break;
+			case TRANSFORM:
+				final AggregationInfoV1 tInfo = new AggregationInfoV1();
+				tInfo.setAggregationStage(stage);
+				tInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
+				tInfo.setDate(getDate(d));
+				info = tInfo;
+				break;
 			}
+			return info;
+		};
+	}
 
-			private CollectionMode getCollectionMode(final Document d) {
-				return Optional.ofNullable(d.getString("system:node:SELECT_MODE:selection"))
-					.map(CollectionMode::valueOf)
-					.orElseGet(() -> Optional.ofNullable(d.getString("collectionMode"))
+	private Function<Document, AggregationInfo> getMapperV2() {
+		return d -> {
+
+			AggregationInfo info = null;
+			final AggregationStage stage = AggregationStage.parse(d.getString("system:wfName"));
+			final boolean success = isCompletedSuccesfully(d);
+
+			switch (stage) {
+
+			case COLLECT:
+				final AggregationInfo cInfo = new AggregationInfo();
+				cInfo.setAggregationStage(stage);
+				cInfo.setCollectionMode(getCollectionMode(d));
+				cInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
+				cInfo.setDate(getDate(d));
+				cInfo.setCompletedSuccessfully(success);
+				info = cInfo;
+				break;
+			case TRANSFORM:
+				final AggregationInfo tInfo = new AggregationInfo();
+				tInfo.setAggregationStage(stage);
+				tInfo.setNumberOfRecords(success ? getNumberOfRecords(d) : 0);
+				tInfo.setDate(getDate(d));
+				tInfo.setCompletedSuccessfully(success);
+				info = tInfo;
+				break;
+			}
+			return info;
+		};
+	}
+
+	private CollectionMode getCollectionMode(final Document d) {
+		return Optional.ofNullable(d.getString("system:node:SELECT_MODE:selection"))
+				.map(CollectionMode::valueOf)
+				.orElseGet(() -> Optional.ofNullable(d.getString("collectionMode"))
 						.map(CollectionMode::valueOf)
 						.orElse(null));
-			}
+	}
 
-			private Integer getNumberOfRecords(final Document d) {
-				final String sinkSize = d.getString("mainlog:sinkSize");
-				final String total = d.getString("mainlog:total");
+	private Integer getNumberOfRecords(final Document d) {
+		final String sinkSize = d.getString("mainlog:sinkSize");
+		final String total = d.getString("mainlog:total");
 
-				if (StringUtils.isNotBlank(sinkSize)) {
-					return Ints.tryParse(sinkSize);
-				} else if (StringUtils.isNotBlank(total)) {
-					return Ints.tryParse(total);
-				} else {
-					return -1;
-				}
-			}
-		};
+		if (StringUtils.isNotBlank(sinkSize)) { return Ints.tryParse(sinkSize); }
+		if (StringUtils.isNotBlank(total)) { return Ints.tryParse(total); }
+		return -1;
 	}
 
 	private String getDate(final Document d) {
@@ -216,7 +287,12 @@ public class MongoLoggerClientImpl implements MongoLoggerClient {
 		return new BasicDBObject(key, value);
 	}
 
-	private Bson queryForAggregationHistory(final String dsId, final String pattern) {
+	@Deprecated
+	private Bson queryForAggregationHistoryV1(final String dsId, final String pattern) {
+		return and(eq("parentDatasourceId", dsId), eq("system:profileFamily", "aggregator"), eq("system:isCompletedSuccessfully", "true"), regex("system:wfName", pattern, "i"));
+	}
+
+	private Bson queryForAggregationHistoryV2(final String dsId, final String pattern) {
 		return and(eq("parentDatasourceId", dsId), eq("system:profileFamily", "aggregator"), regex("system:wfName", pattern, "i"));
 	}
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/ObjectStoreClient.java

@@ -1,9 +1,9 @@
 package eu.dnetlib.openaire.dsm.dao;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
 
 public interface ObjectStoreClient {
 
-	Long getObjectStoreSize(final String objectStoreId) throws DsmException;
+	Long getObjectStoreSize(final String objectStoreId) throws DsmApiException;
 
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/ObjectStoreClientImpl.java

@@ -1,8 +1,5 @@
 package eu.dnetlib.openaire.dsm.dao;
 
-import eu.dnetlib.data.objectstore.rmi.ObjectStoreService;
-import eu.dnetlib.data.objectstore.rmi.ObjectStoreServiceException;
-import eu.dnetlib.enabling.datasources.common.DsmException;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
@@ -10,6 +7,10 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
 import org.springframework.stereotype.Component;
 
+import eu.dnetlib.data.objectstore.rmi.ObjectStoreService;
+import eu.dnetlib.data.objectstore.rmi.ObjectStoreServiceException;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+
 @Component
 @ConditionalOnProperty(value = "openaire.exporter.enable.dsm", havingValue = "true")
 public class ObjectStoreClientImpl implements ObjectStoreClient {
@@ -20,17 +21,15 @@ public class ObjectStoreClientImpl implements ObjectStoreClient {
 	private ObjectStoreService objectStoreService;
 
 	@Override
-	public Long getObjectStoreSize(final String objectStoreId) throws DsmException {
+	public Long getObjectStoreSize(final String objectStoreId) throws DsmApiException {
 		log.debug("get size for objectStore " + objectStoreId);
-		if (StringUtils.isBlank(objectStoreId)) {
-			return 0L;
-		}
+		if (StringUtils.isBlank(objectStoreId)) { return 0L; }
 		try {
 			final long size = objectStoreService.getSize(objectStoreId);
 			log.debug("got objectStore size: " + size);
 			return size;
-		} catch (ObjectStoreServiceException e) {
-			throw new DsmException("unable to get size for objectStore " + objectStoreId);
+		} catch (final ObjectStoreServiceException e) {
+			throw new DsmApiException("unable to get size for objectStore " + objectStoreId);
 		}
 	}
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/ResponseUtils.java

@@ -5,16 +5,16 @@ import java.util.Queue;
 
 import com.google.common.collect.Lists;
 
-import eu.dnetlib.openaire.dsm.domain.ApiDetails;
-import eu.dnetlib.openaire.dsm.domain.ApiDetailsResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetails;
-import eu.dnetlib.openaire.dsm.domain.DatasourceInfo;
 import eu.dnetlib.openaire.dsm.domain.DatasourceSearchResponse;
-import eu.dnetlib.openaire.dsm.domain.DatasourceSnippetExtended;
-import eu.dnetlib.openaire.dsm.domain.DatasourceSnippetResponse;
-import eu.dnetlib.openaire.dsm.domain.Header;
-import eu.dnetlib.openaire.dsm.domain.SimpleResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetailsResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceSnippetExtended;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceSnippetResponse;
+import eu.dnetlib.openaire.exporter.model.dsm.Header;
+import eu.dnetlib.openaire.exporter.model.dsm.SimpleResponse;
 
 public class ResponseUtils {
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/VocabularyClient.java

@@ -1,13 +1,13 @@
 package eu.dnetlib.openaire.dsm.dao;
 
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.openaire.vocabularies.Vocabulary;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Vocabulary;
 
 public interface VocabularyClient {
 
-	Vocabulary getCountries() throws DsmException;
+	Vocabulary getCountries() throws DsmApiException;
 
-	Vocabulary getDatasourceTypologies() throws DsmException;
+	Vocabulary getDatasourceTypologies() throws DsmApiException;
 
 	void dropCache();
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/VocabularyClientImpl.java

@@ -1,8 +1,5 @@
 package eu.dnetlib.openaire.dsm.dao;
 
-import eu.dnetlib.DnetOpenaireExporterProperties;
-import eu.dnetlib.enabling.datasources.common.DsmException;
-import eu.dnetlib.openaire.vocabularies.Vocabulary;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -13,6 +10,10 @@ import org.springframework.scheduling.annotation.Scheduled;
 import org.springframework.stereotype.Component;
 import org.springframework.web.client.RestTemplate;
 
+import eu.dnetlib.DnetOpenaireExporterProperties;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Vocabulary;
+
 /**
  * Created by claudio on 15/09/2017.
  */
@@ -26,24 +27,22 @@ public class VocabularyClientImpl implements VocabularyClient {
 
 	@Override
 	@Cacheable("vocabularies-cache")
-	public Vocabulary getCountries() throws DsmException {
+	public Vocabulary getCountries() throws DsmApiException {
 		return _getVocabulary(config.getVocabularies().getCountriesEndpoint(), Vocabulary.class);
 	}
 
 	@Override
 	@Cacheable("vocabularies-cache")
-	public Vocabulary getDatasourceTypologies() throws DsmException {
+	public Vocabulary getDatasourceTypologies() throws DsmApiException {
 		return _getVocabulary(config.getVocabularies().getDatasourceTypologiesEndpoint(), Vocabulary.class);
 	}
 
-	private <T> T _getVocabulary(final String endpoint, Class<T> clazz) throws DsmException {
+	private <T> T _getVocabulary(final String endpoint, final Class<T> clazz) throws DsmApiException {
 		final RestTemplate rt = new RestTemplate();
 		log.info("get vocabulary from " + endpoint);
 		final ResponseEntity<T> rsp = rt.getForEntity(endpoint, clazz);
 
-		if (!rsp.getStatusCode().is2xxSuccessful()) {
-			throw new DsmException(rsp.getStatusCodeValue(), "unable to read content from " + endpoint);
-		}
+		if (!rsp.getStatusCode().is2xxSuccessful()) { throw new DsmApiException(rsp.getStatusCodeValue(), "unable to read content from " + endpoint); }
 
 		return rsp.getBody();
 	}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/dao/utils/DsmMappingUtils.java

@@ -14,14 +14,14 @@ import org.springframework.beans.BeanWrapperImpl;
 import com.fasterxml.jackson.databind.ObjectMapper;
 
 import eu.dnetlib.data.transform.xml.AbstractDNetXsltFunctions;
-import eu.dnetlib.openaire.dsm.domain.ApiDetails;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetails;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetailsUpdate;
-import eu.dnetlib.openaire.dsm.domain.DatasourceSnippetExtended;
-import eu.dnetlib.openaire.dsm.domain.OrganizationDetails;
 import eu.dnetlib.openaire.dsm.domain.db.ApiDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceDbEntry;
 import eu.dnetlib.openaire.dsm.domain.db.OrganizationDbEntry;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetails;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetailsUpdate;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceSnippetExtended;
+import eu.dnetlib.openaire.exporter.model.dsm.OrganizationDetails;
 
 public class DsmMappingUtils {
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/domain/CollectionInfo.java

@@ -1,24 +0,0 @@
-package eu.dnetlib.openaire.dsm.domain;
-
-import com.fasterxml.jackson.annotation.JsonAutoDetect;
-
-import eu.dnetlib.enabling.datasources.common.AggregationInfo;
-import io.swagger.v3.oas.annotations.media.Schema;
-
-/**
- * Created by claudio on 29/11/2016.
- */
-@JsonAutoDetect
-public class CollectionInfo extends AggregationInfo {
-
-	private CollectionMode collectionMode;
-
-	public CollectionMode getCollectionMode() {
-		return collectionMode;
-	}
-
-	public void setCollectionMode(final CollectionMode collectionMode) {
-		this.collectionMode = collectionMode;
-	}
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/domain/DatasourceResponse.java

@@ -5,6 +5,8 @@ import java.util.List;
 import com.fasterxml.jackson.annotation.JsonAutoDetect;
 import com.google.common.collect.Lists;
 
+import eu.dnetlib.openaire.exporter.model.dsm.Response;
+
 @JsonAutoDetect
 public class DatasourceResponse<T> extends Response {
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/domain/DatasourceSearchResponse.java

@@ -4,6 +4,9 @@ import java.util.List;
 
 import com.fasterxml.jackson.annotation.JsonAutoDetect;
 
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.Response;
+
 @JsonAutoDetect
 public class DatasourceSearchResponse extends Response {
 

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/dsm/domain/TransformationInfo.java

@@ -1,13 +0,0 @@
-package eu.dnetlib.openaire.dsm.domain;
-
-import com.fasterxml.jackson.annotation.JsonAutoDetect;
-
-import eu.dnetlib.enabling.datasources.common.AggregationInfo;
-
-/**
- * Created by claudio on 29/11/2016.
- */
-@JsonAutoDetect
-public class TransformationInfo extends AggregationInfo {
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/FunderDao.java

@@ -1,43 +0,0 @@
-package eu.dnetlib.openaire.funders;
-
-import java.util.List;
-import java.util.stream.Collectors;
-
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
-import org.springframework.data.domain.PageRequest;
-import org.springframework.stereotype.Component;
-
-import eu.dnetlib.openaire.funders.domain.ConversionUtils;
-import eu.dnetlib.openaire.funders.domain.ExtendedFunderDetails;
-import eu.dnetlib.openaire.funders.domain.FunderDetails;
-
-@Component
-@ConditionalOnProperty(value = "openaire.exporter.enable.funders", havingValue = "true")
-public class FunderDao {
-
-	@Autowired
-	private FunderRepository funderRepository;
-
-	public ExtendedFunderDetails getExtendedFunderDetails(final String funderId) throws FundersApiException {
-		return ConversionUtils
-			.asExtendedFunderDetails(funderRepository.findById(funderId).orElseThrow(() -> new FundersApiException("Funder not found. ID:  " + funderId)));
-	}
-
-	public List<FunderDetails> listFunderDetails(final int page, final int size) throws FundersApiException {
-		return funderRepository.findAll(PageRequest.of(page, size))
-			.getContent()
-			.stream()
-			.map(ConversionUtils::asFunderDetails)
-			.collect(Collectors.toList());
-	}
-
-	public List<String> listFunderIds(final int page, final int size) throws FundersApiException {
-		return funderRepository.findAll(PageRequest.of(page, size))
-			.getContent()
-			.stream()
-			.map(f -> f.getId())
-			.collect(Collectors.toList());
-	}
-
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/FunderRepository.java

@@ -1,10 +1,11 @@
 package eu.dnetlib.openaire.funders;
 
-import eu.dnetlib.openaire.funders.domain.db.FunderDbEntry;
-import org.springframework.data.jpa.repository.JpaRepository;
 import org.springframework.stereotype.Repository;
 
+import eu.dnetlib.openaire.common.ReadOnlyRepository;
+import eu.dnetlib.openaire.funders.domain.db.FunderDbEntry;
+
 @Repository
-public interface FunderRepository extends JpaRepository<FunderDbEntry, String> {
+public interface FunderRepository extends ReadOnlyRepository<FunderDbEntry, String> {
 
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/FunderService.java

@@ -0,0 +1,183 @@
+package eu.dnetlib.openaire.funders;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.FilenameFilter;
+import java.time.LocalDate;
+import java.time.format.DateTimeFormatter;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import javax.annotation.PostConstruct;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Component;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+
+import eu.dnetlib.openaire.dsm.dao.MongoLoggerClient;
+import eu.dnetlib.openaire.exporter.exceptions.DsmApiException;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationInfo;
+import eu.dnetlib.openaire.exporter.model.dsm.AggregationStage;
+import eu.dnetlib.openaire.funders.domain.db.FunderDatasource;
+import eu.dnetlib.openaire.funders.domain.db.FunderDbEntry;
+import eu.dnetlib.openaire.funders.domain.db.FunderPid;
+
+@Component
+@ConditionalOnProperty(value = "openaire.exporter.enable.funders", havingValue = "true")
+public class FunderService {
+
+	private static final String TEMP_FILE_SUFFIX = ".funds.tmp";
+	private static final String SEPARATOR = "@=@";
+
+	@Autowired
+	private FunderRepository funderRepository;
+
+	@Autowired
+	private MongoLoggerClient mongoLoggerClient;
+
+	private File tempDir;
+
+	private File tempFile;
+
+	private final DateTimeFormatter DATEFORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd");
+
+	private static final Log log = LogFactory.getLog(FunderService.class);
+
+	@PostConstruct
+	public void init() {
+
+		tempDir = new File(System.getProperty("java.io.tmpdir", "/tmp"));
+
+		for (final File f : tempDir.listFiles((FilenameFilter) (dir, name) -> name.endsWith(TEMP_FILE_SUFFIX))) {
+			deleteFile(f);
+		}
+
+		new Thread(this::updateFunders).start();
+	}
+
+	private void deleteFile(final File f) {
+		if (f != null && f.exists()) {
+			log.info("Deleting file: " + f.getAbsolutePath());
+			f.delete();
+		}
+	}
+
+	@Scheduled(cron = "${openaire.exporter.funders.cron}")
+	public void updateFunders() {
+		try {
+			final ObjectMapper mapper = new ObjectMapper();
+			mapper.registerModule(new JavaTimeModule());
+			mapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
+
+			final File tmp = File.createTempFile("funders-api-", TEMP_FILE_SUFFIX, tempDir);
+
+			log.info("Generating funders file: " + tmp.getAbsolutePath());
+
+			try (final FileWriter writer = new FileWriter(tmp)) {
+				writer.write("[");
+				boolean first = true;
+				for (final FunderDbEntry funder : funderRepository.findAll()) {
+					log.info("  - adding: " + funder.getId());
+
+					// THIS PATCH IS NECESSARY FOR COMPATIBILITY WITH POSTGRES 9.3 (PARTIAL SUPPORT OF THE JSON LIBRARY)
+
+					final List<FunderDatasource> datasources = Arrays.stream(funder.getDatasourcesPostgres())
+							.filter(Objects::nonNull)
+							.map(s -> s.split(SEPARATOR))
+							.filter(arr -> arr.length == 3)
+							.map(arr -> {
+								final FunderDatasource ds = new FunderDatasource();
+								ds.setId(arr[0].trim());
+								ds.setName(arr[1].trim());
+								ds.setType(arr[2].trim());
+								return ds;
+							})
+							.filter(ds -> StringUtils.isNotBlank(ds.getId()))
+							.collect(Collectors.toList());
+
+					funder.setDatasources(datasources);
+
+					final List<FunderPid> pids = Arrays.stream(funder.getPidsPostgres())
+							.filter(Objects::nonNull)
+							.map(s -> s.split(SEPARATOR))
+							.filter(arr -> arr.length == 2)
+							.map(arr -> {
+								final FunderPid pid = new FunderPid();
+								pid.setType(arr[0].trim());
+								pid.setValue(arr[1].trim());
+								return pid;
+							})
+							.filter(pid -> StringUtils.isNotBlank(pid.getValue()))
+							.collect(Collectors.toList());
+
+					funder.setPids(pids);
+
+					// END PATCH
+
+					addAggregationHistory(funder);
+
+					if (first) {
+						first = false;
+					} else {
+						writer.write(",");
+					}
+					writer.write(mapper.writeValueAsString(funder));
+				}
+				writer.write("]");
+				log.info("Publish funders file: " + tmp.getAbsolutePath());
+
+				deleteFile(tempFile);
+				setTempFile(tmp);
+			}
+		} catch (final Throwable e) {
+			log.error("Error generating funders file", e);
+			throw new RuntimeException("Error generating funders file", e);
+		}
+	}
+
+	private void addAggregationHistory(final FunderDbEntry funder) {
+
+		final List<LocalDate> dates = funder.getDatasources()
+				.stream()
+				.map(FunderDatasource::getId)
+				.map(id -> {
+					try {
+						return mongoLoggerClient.getAggregationHistoryV2(id);
+					} catch (final DsmApiException e) {
+						log.error("Error retrieving the aggregation history", e);
+						throw new RuntimeException("Error retrieving the aggregation history", e);
+					}
+				})
+				.flatMap(List::stream)
+				.filter(AggregationInfo::isCompletedSuccessfully)
+				.filter(info -> info.getAggregationStage() == AggregationStage.TRANSFORM)
+				.map(AggregationInfo::getDate)
+				.distinct()
+				.map(s -> LocalDate.parse(s, DATEFORMATTER))
+				.sorted(Comparator.reverseOrder())
+				.limit(10)
+				.collect(Collectors.toList());
+
+		funder.setAggregationDates(dates);
+	}
+
+	public File getTempFile() {
+		return tempFile;
+	}
+
+	public void setTempFile(final File tempFile) {
+		this.tempFile = tempFile;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/FundersApiController.java

@@ -1,20 +1,25 @@
 package eu.dnetlib.openaire.funders;
 
-import java.util.List;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
 
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.commons.io.IOUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.http.MediaType;
 import org.springframework.web.bind.annotation.CrossOrigin;
-import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RequestMethod;
 import org.springframework.web.bind.annotation.RestController;
 
 import eu.dnetlib.openaire.common.AbstractExporterController;
-import eu.dnetlib.openaire.funders.domain.ExtendedFunderDetails;
-import eu.dnetlib.openaire.funders.domain.FunderDetails;
+import eu.dnetlib.openaire.exporter.exceptions.FundersApiException;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.responses.ApiResponses;
@@ -22,59 +27,48 @@ import io.swagger.v3.oas.annotations.tags.Tag;
 
 @RestController
 @CrossOrigin(origins = {
-	"*"
+		"*"
 })
 @ConditionalOnProperty(value = "openaire.exporter.enable.funders", havingValue = "true")
 @Tag(name = "OpenAIRE funders API", description = "the OpenAIRE funders API")
 public class FundersApiController extends AbstractExporterController {
 
+	@Autowired
+	private FunderService service;
+
 	private static final Log log = LogFactory.getLog(FundersApiController.class);
 
-	@Autowired
-	private FunderDao fDao;
-
 	@RequestMapping(value = "/funders", produces = {
-		"application/json"
+			"application/json"
 	}, method = RequestMethod.GET)
-	@Operation(summary = "get basic information about funders", description = "basic information about funders: id, name, shortname, last update date, registration date")
+	@Operation(summary = "get all funders", description = "get all funders")
 	@ApiResponses(value = {
-		@ApiResponse(responseCode = "200", description = "OK"),
-		@ApiResponse(responseCode = "500", description = "unexpected error")
+			@ApiResponse(responseCode = "200", description = "OK"),
+			@ApiResponse(responseCode = "500", description = "unexpected error")
 	})
-	public List<FunderDetails> getFunders(
-		@PathVariable final int page,
-		@PathVariable final int size) throws FundersApiException {
+	public void getFunders(final HttpServletResponse res) throws FundersApiException {
 
-		return fDao.listFunderDetails(page, size);
-	}
+		res.setContentType(MediaType.APPLICATION_JSON_VALUE);
 
-	@RequestMapping(value = "/funder/{id}", produces = {
-		"application/json"
-	}, method = RequestMethod.GET)
-	@Operation(summary = "get the funder details", description = "complete funder information")
-	@ApiResponses(value = {
-		@ApiResponse(responseCode = "200", description = "OK"),
-		@ApiResponse(responseCode = "500", description = "unexpected error")
-	})
-	public ExtendedFunderDetails getFunderDetails(
-		@PathVariable final String id) throws FundersApiException {
+		final File file = service.getTempFile();
 
-		return fDao.getExtendedFunderDetails(id);
-	}
+		if (file == null) {
+			log.error("Missing temp file (NULL)");
+			throw new FundersApiException("Missing temp file (NULL)");
+		}
 
-	@RequestMapping(value = "/funder/ids", produces = {
-		"application/json"
-	}, method = RequestMethod.GET)
-	@Operation(summary = "get the list of funder ids", description = "get the list of funder ids")
-	@ApiResponses(value = {
-		@ApiResponse(responseCode = "200", description = "OK"),
-		@ApiResponse(responseCode = "500", description = "unexpected error")
-	})
-	public List<String> getFunderIds(
-		@PathVariable final int page,
-		@PathVariable final int size) throws FundersApiException {
+		if (!file.exists()) {
+			log.error("Missing temp file " + service.getTempFile());
+			throw new FundersApiException("Missing temp file " + service.getTempFile());
+		}
 
-		return fDao.listFunderIds(page, size);
+		try (final InputStream in = new FileInputStream(file); OutputStream out = res.getOutputStream()) {
+			IOUtils.copy(in, out);
+			return;
+		} catch (final Exception e) {
+			log.error("Error reading file " + service.getTempFile(), e);
+			throw new FundersApiException("Error reading file " + service.getTempFile(), e);
+		}
 	}
 
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/ConversionUtils.java

@@ -1,88 +0,0 @@
-package eu.dnetlib.openaire.funders.domain;
-
-import java.io.StringReader;
-import java.util.stream.Collectors;
-
-import eu.dnetlib.openaire.funders.domain.db.FunderDbEntry;
-import eu.dnetlib.openaire.funders.domain.db.FundingPathDbEntry;
-import org.dom4j.Document;
-import org.dom4j.DocumentException;
-import org.dom4j.io.SAXReader;
-
-import static org.apache.commons.lang3.StringUtils.*;
-
-public class ConversionUtils {
-
-	public static final String SEPARATOR = "::";
-
-	public static FunderDetails asFunderDetails(final FunderDbEntry fdb) {
-		final FunderDetails f = new FunderDetails();
-
-		f.setId(fdb.getId());
-		f.setName(fdb.getName());
-		f.setShortname(fdb.getShortname());
-		f.setJurisdiction(fdb.getJurisdiction());
-		f.setRegistrationDate(fdb.getRegistrationdate());
-		f.setLastUpdateDate(fdb.getLastupdatedate());
-
-		return f;
-	}
-
-	public static ExtendedFunderDetails asExtendedFunderDetails(final FunderDbEntry fdb) {
-		final ExtendedFunderDetails f = new ExtendedFunderDetails(asFunderDetails(fdb));
-
-		if (fdb.getFundingpaths() != null) {
-			f.setFundingStreams(
-					fdb.getFundingpaths().stream()
-						.map(ConversionUtils::asFundingStream)
-						.collect(Collectors.toList()));
-		}
-
-		return f;
-	}
-
-	private static FundingStream asFundingStream(final FundingPathDbEntry pathDbEntry) {
-		final FundingStream f = new FundingStream();
-
-		try {
-			final Document xml = new SAXReader().read(new StringReader(pathDbEntry.getPath()));
-
-			for(int i=2;i>=0;i--) {
-				if (hasFundingLevel(i, xml)) {
-					f.setId(getId(i, xml));
-					f.setName(getName(i, xml));
-					break;
-				}
-			}
-
-			if (isBlank(f.getId()) && isNoneBlank(xml.valueOf("//funder/id/text()"))) {
-				f.setId(xml.valueOf("//funder/shortname/text()"));
-				f.setName(xml.valueOf("//funder/name/text()"));
-			}
-
-			if (isBlank(f.getId())) {
-				throw new IllegalStateException("invalid funding path:\n" + xml.asXML());
-			}
-
-			return f;
-
-		} catch (DocumentException e) {
-			throw new IllegalStateException("unable to parse funding path:\n" + pathDbEntry.getPath());
-		}
-	}
-
-	private static String getName(int level, final Document d) {
-		return d.valueOf(String.format("//funding_level_%s/description/text()", level));
-	}
-
-	private static String getId(int level, final Document d) {
-		return substringAfter(
-				d.valueOf(
-					String.format("//funding_level_%s/id/text()", level)
-				), SEPARATOR);
-	}
-
-	private static boolean hasFundingLevel(int level, Document d) {
-		return isNotBlank(getId(level, d));
-	}
-}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/db/FunderDatasource.java

@@ -0,0 +1,39 @@
+package eu.dnetlib.openaire.funders.domain.db;
+
+import java.io.Serializable;
+
+public class FunderDatasource implements Serializable {
+
+	private static final long serialVersionUID = 2145493560459874509L;
+
+	private String id;
+
+	private String name;
+
+	private String type;
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(final String id) {
+		this.id = id;
+	}
+
+	public String getName() {
+		return name;
+	}
+
+	public void setName(final String name) {
+		this.name = name;
+	}
+
+	public String getType() {
+		return type;
+	}
+
+	public void setType(final String type) {
+		this.type = type;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/db/FunderDbEntry.java

@@ -1,38 +1,75 @@
 package eu.dnetlib.openaire.funders.domain.db;
 
-import java.sql.Date;
-import java.util.Set;
-import javax.persistence.*;
+import java.io.Serializable;
+import java.time.LocalDate;
+import java.util.ArrayList;
+import java.util.List;
 
-import eu.dnetlib.openaire.dsm.domain.db.IdentityDbEntry;
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.Table;
+import javax.persistence.Transient;
+
+import org.hibernate.annotations.Type;
+import org.hibernate.annotations.TypeDef;
+import org.hibernate.annotations.TypeDefs;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.vladmihalcea.hibernate.type.array.StringArrayType;
 
 @Entity
-@Table(name = "funders")
-public class FunderDbEntry {
+@Table(name = "funders_view")
+@TypeDefs({
+		@TypeDef(name = "string-array", typeClass = StringArrayType.class)
+})
+public class FunderDbEntry implements Serializable {
+
+	private static final long serialVersionUID = 1290088460508203016L;
 
 	@Id
+	@Column(name = "id")
 	private String id;
-	private String name;
-	private String shortname;
-	private String jurisdiction;
-	private String websiteurl;
-	private String policy;
-	private Date registrationdate;
-	private Date lastupdatedate;
 
-	@OneToMany(
-			cascade = { CascadeType.PERSIST, CascadeType.MERGE },
-			fetch = FetchType.LAZY )
-	@JoinTable(
-			name = "funder_identity",
-			joinColumns = @JoinColumn(name = "funder"),
-			inverseJoinColumns = @JoinColumn(name = "pid"))
-	private Set<IdentityDbEntry> pids;
+	@Column(name = "legalshortname")
+	private String legalShortName;
 
-	@OneToMany(mappedBy = "funderid", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
-	private Set<FundingPathDbEntry> fundingpaths;
+	@Column(name = "legalname")
+	private String legalName;
 
-	public FunderDbEntry() {}
+	@Column(name = "websiteurl")
+	private String websiteUrl;
+
+	@Column(name = "logourl")
+	private String logoUrl;
+
+	@Column(name = "country")
+	private String country;
+
+	@Column(name = "registrationdate")
+	private LocalDate registrationDate;
+
+	@Column(name = "registered")
+	private Boolean registered;
+
+	@JsonIgnore
+	@Type(type = "string-array")
+	@Column(name = "pids", columnDefinition = "text[]")
+	private String[] pidsPostgres;
+
+	@Transient
+	private List<FunderPid> pids = new ArrayList<FunderPid>();
+
+	@JsonIgnore
+	@Type(type = "string-array")
+	@Column(name = "datasources", columnDefinition = "text[]")
+	private String[] datasourcesPostgres;
+
+	@Transient
+	private List<FunderDatasource> datasources = new ArrayList<FunderDatasource>();
+
+	@Transient
+	private List<LocalDate> aggregationDates;
 
 	public String getId() {
 		return id;
@@ -42,75 +79,100 @@ public class FunderDbEntry {
 		this.id = id;
 	}
 
-	public String getName() {
-		return name;
+	public String getLegalShortName() {
+		return legalShortName;
 	}
 
-	public void setName(final String name) {
-		this.name = name;
+	public void setLegalShortName(final String legalShortName) {
+		this.legalShortName = legalShortName;
 	}
 
-	public String getShortname() {
-		return shortname;
+	public String getLegalName() {
+		return legalName;
 	}
 
-	public void setShortname(final String shortname) {
-		this.shortname = shortname;
+	public void setLegalName(final String legalName) {
+		this.legalName = legalName;
 	}
 
-	public String getJurisdiction() {
-		return jurisdiction;
+	public String getWebsiteUrl() {
+		return websiteUrl;
 	}
 
-	public void setJurisdiction(final String jurisdiction) {
-		this.jurisdiction = jurisdiction;
+	public void setWebsiteUrl(final String websiteUrl) {
+		this.websiteUrl = websiteUrl;
 	}
 
-	public String getWebsiteurl() {
-		return websiteurl;
+	public String getLogoUrl() {
+		return logoUrl;
 	}
 
-	public void setWebsiteurl(final String websiteurl) {
-		this.websiteurl = websiteurl;
+	public void setLogoUrl(final String logoUrl) {
+		this.logoUrl = logoUrl;
 	}
 
-	public String getPolicy() {
-		return policy;
+	public String getCountry() {
+		return country;
 	}
 
-	public void setPolicy(final String policy) {
-		this.policy = policy;
+	public void setCountry(final String country) {
+		this.country = country;
 	}
 
-	public Date getRegistrationdate() {
-		return registrationdate;
+	public LocalDate getRegistrationDate() {
+		return registrationDate;
 	}
 
-	public void setRegistrationdate(final Date registrationdate) {
-		this.registrationdate = registrationdate;
+	public void setRegistrationDate(final LocalDate registrationDate) {
+		this.registrationDate = registrationDate;
 	}
 
-	public Date getLastupdatedate() {
-		return lastupdatedate;
+	public Boolean getRegistered() {
+		return registered;
 	}
 
-	public void setLastupdatedate(final Date lastupdatedate) {
-		this.lastupdatedate = lastupdatedate;
+	public void setRegistered(final Boolean registered) {
+		this.registered = registered;
 	}
 
-	public Set<IdentityDbEntry> getPids() {
+	public String[] getPidsPostgres() {
+		return pidsPostgres;
+	}
+
+	public void setPidsPostgres(final String[] pidsPostgres) {
+		this.pidsPostgres = pidsPostgres;
+	}
+
+	public List<FunderPid> getPids() {
 		return pids;
 	}
 
-	public void setPids(final Set<IdentityDbEntry> pids) {
+	public void setPids(final List<FunderPid> pids) {
 		this.pids = pids;
 	}
 
-	public Set<FundingPathDbEntry> getFundingpaths() {
-		return fundingpaths;
+	public String[] getDatasourcesPostgres() {
+		return datasourcesPostgres;
 	}
 
-	public void setFundingpaths(final Set<FundingPathDbEntry> fundingpaths) {
-		this.fundingpaths = fundingpaths;
+	public void setDatasourcesPostgres(final String[] datasourcesPostgres) {
+		this.datasourcesPostgres = datasourcesPostgres;
 	}
+
+	public List<FunderDatasource> getDatasources() {
+		return datasources;
+	}
+
+	public void setDatasources(final List<FunderDatasource> datasources) {
+		this.datasources = datasources;
+	}
+
+	public List<LocalDate> getAggregationDates() {
+		return aggregationDates;
+	}
+
+	public void setAggregationDates(final List<LocalDate> aggregationDates) {
+		this.aggregationDates = aggregationDates;
+	}
+
 }

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/db/FunderPid.java

@@ -0,0 +1,29 @@
+package eu.dnetlib.openaire.funders.domain.db;
+
+import java.io.Serializable;
+
+public class FunderPid implements Serializable {
+
+	private static final long serialVersionUID = 2145493560459874509L;
+
+	private String type;
+
+	private String value;
+
+	public String getType() {
+		return type;
+	}
+
+	public void setType(final String type) {
+		this.type = type;
+	}
+
+	public String getValue() {
+		return value;
+	}
+
+	public void setValue(final String value) {
+		this.value = value;
+	}
+
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/db/OldFunderDbEntry.java

@@ -0,0 +1,118 @@
+package eu.dnetlib.openaire.funders.domain.db;
+
+import java.sql.Date;
+import java.util.Set;
+
+import javax.persistence.CascadeType;
+import javax.persistence.FetchType;
+import javax.persistence.Id;
+import javax.persistence.JoinColumn;
+import javax.persistence.JoinTable;
+import javax.persistence.OneToMany;
+
+import eu.dnetlib.openaire.dsm.domain.db.IdentityDbEntry;
+
+// @Entity
+// @Table(name = "funders")
+@Deprecated
+public class OldFunderDbEntry {
+
+	@Id
+	private String id;
+	private String name;
+	private String shortname;
+	private String jurisdiction;
+	private String websiteurl;
+	private String policy;
+	private Date registrationdate;
+	private Date lastupdatedate;
+
+	@OneToMany(cascade = { CascadeType.PERSIST, CascadeType.MERGE }, fetch = FetchType.LAZY)
+	@JoinTable(name = "funder_identity", joinColumns = @JoinColumn(name = "funder"), inverseJoinColumns = @JoinColumn(name = "pid"))
+	private Set<IdentityDbEntry> pids;
+
+	@OneToMany(mappedBy = "funderid", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
+	private Set<OldFundingPathDbEntry> fundingpaths;
+
+	public OldFunderDbEntry() {}
+
+	public String getId() {
+		return id;
+	}
+
+	public void setId(final String id) {
+		this.id = id;
+	}
+
+	public String getName() {
+		return name;
+	}
+
+	public void setName(final String name) {
+		this.name = name;
+	}
+
+	public String getShortname() {
+		return shortname;
+	}
+
+	public void setShortname(final String shortname) {
+		this.shortname = shortname;
+	}
+
+	public String getJurisdiction() {
+		return jurisdiction;
+	}
+
+	public void setJurisdiction(final String jurisdiction) {
+		this.jurisdiction = jurisdiction;
+	}
+
+	public String getWebsiteurl() {
+		return websiteurl;
+	}
+
+	public void setWebsiteurl(final String websiteurl) {
+		this.websiteurl = websiteurl;
+	}
+
+	public String getPolicy() {
+		return policy;
+	}
+
+	public void setPolicy(final String policy) {
+		this.policy = policy;
+	}
+
+	public Date getRegistrationdate() {
+		return registrationdate;
+	}
+
+	public void setRegistrationdate(final Date registrationdate) {
+		this.registrationdate = registrationdate;
+	}
+
+	public Date getLastupdatedate() {
+		return lastupdatedate;
+	}
+
+	public void setLastupdatedate(final Date lastupdatedate) {
+		this.lastupdatedate = lastupdatedate;
+	}
+
+	public Set<IdentityDbEntry> getPids() {
+		return pids;
+	}
+
+	public void setPids(final Set<IdentityDbEntry> pids) {
+		this.pids = pids;
+	}
+
+	public Set<OldFundingPathDbEntry> getFundingpaths() {
+		return fundingpaths;
+	}
+
+	public void setFundingpaths(final Set<OldFundingPathDbEntry> fundingpaths) {
+		this.fundingpaths = fundingpaths;
+	}
+}

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/funders/domain/db/OldFundingPathDbEntry.java

@@ -1,12 +1,15 @@
 package eu.dnetlib.openaire.funders.domain.db;
 
-import javax.persistence.*;
+import javax.persistence.Column;
+import javax.persistence.Id;
+import javax.persistence.Transient;
 
 import com.fasterxml.jackson.annotation.JsonIgnore;
 
-@Entity
-@Table(name = "fundingpaths")
-public class FundingPathDbEntry {
+// @Entity
+// @Table(name = "fundingpaths")
+@Deprecated
+public class OldFundingPathDbEntry {
 
 	@JsonIgnore
 	@Transient
@@ -28,7 +31,7 @@ public class FundingPathDbEntry {
 
 	private String funderid;
 
-	public FundingPathDbEntry() {}
+	public OldFundingPathDbEntry() {}
 
 	public String getDnetresourceidentifier() {
 		return dnetresourceidentifier;

apps/dnet-exporter-api/src/main/java/eu/dnetlib/openaire/info/JdbcInfoDao.java

@@ -8,6 +8,7 @@ public interface JdbcInfoDao {
 		claim_load_date,
 		oaf_load_date,
 		odf_load_date,
+		db_load_date,
 		inference_date,
 		stats_update_date,
 		crossref_update_date,

apps/dnet-exporter-api/src/main/resources/application.properties

@@ -22,12 +22,12 @@ management.endpoints.web.base-path                  =   /
 management.endpoints.web.path-mapping.prometheus    =   metrics
 management.endpoints.web.path-mapping.health        =   health
 
-
 # ENABLE / DISABLE CONTROLLERS
 openaire.exporter.enable.dsm            =   true
 openaire.exporter.enable.community      =   true
+openaire.exporter.enable.community.import =   false
 openaire.exporter.enable.context        =   true
-openaire.exporter.enable.funders        =   false
+openaire.exporter.enable.funders        =   true
 openaire.exporter.enable.project        =   true
 openaire.exporter.enable.info           =   true
 
@@ -36,3 +36,4 @@ openaire.exporter.cache.ttl             =   43200000
 
 maven.pom.path = /META-INF/maven/eu.dnetlib.dhp/dnet-exporter-api/effective-pom.xml
 
+openaire.exporter.funders.cron = 0 30 2 * * *

apps/dnet-exporter-api/src/main/resources/global.properties

@@ -1,7 +1,8 @@
+services.is.protocol                =   http
 services.is.host                    =   localhost
 services.is.port                    =   8280
-services.is.protocol                =   http
 services.is.context                 =   app
+
 services.is.baseurl                 =   ${services.is.protocol}://${services.is.host}:${services.is.port}/${services.is.context}/services
 
 openaire.exporter.isLookupUrl           =   ${services.is.baseurl}/isLookUp
@@ -16,6 +17,7 @@ openaire.exporter.cxfClientReceiveTimeout    =   120000
 
 # JDBC
 openaire.exporter.jdbc.url          =   jdbc:postgresql://localhost:5432/dnet_openaireplus
+#openaire.exporter.jdbc.url          =   jdbc:postgresql://localhost:5432/dev_openaire_8280
 openaire.exporter.jdbc.user         =   dnetapi
 openaire.exporter.jdbc.pwd          =   dnetPwd
 openaire.exporter.jdbc.minIdle      =   1

apps/dnet-exporter-api/src/main/resources/global.properties.locale

@@ -0,0 +1,111 @@
+#services.is.host                    =   localhost
+services.is.host                    =   dev-openaire.d4science.org
+#services.is.port                    =   8280
+services.is.port                    =   443
+#services.is.protocol                =   http
+services.is.protocol                =   https
+#services.is.context                 =   app
+services.is.context                 =   is
+#services.is.baseurl                 =   ${services.is.protocol}://${services.is.host}:${services.is.port}/${services.is.context}/services
+services.is.baseurl                 =   ${services.is.protocol}://${services.is.host}:${services.is.port}/${services.is.context}/services
+
+openaire.exporter.isLookupUrl           =   ${services.is.baseurl}/isLookUp
+openaire.exporter.objectStoreServiceUrl =   ${services.is.baseurl}/objectStore
+openaire.exporter.isRegistryServiceUrl  =   ${services.is.baseurl}/isRegistry
+
+openaire.exporter.requestWorkers    =   10
+openaire.exporter.requestTimeout    =   10
+
+openaire.exporter.cxfClientConnectTimeout    =   60000
+openaire.exporter.cxfClientReceiveTimeout    =   120000
+
+# JDBC
+#openaire.exporter.jdbc.url          =   jdbc:postgresql://localhost:5432/dnet_openaire
+openaire.exporter.jdbc.url          =   jdbc:postgresql://localhost:5432/dev_openaire_8280
+openaire.exporter.jdbc.user         =   dnetapi
+openaire.exporter.jdbc.pwd          =   dnetPwd
+openaire.exporter.jdbc.minIdle      =   1
+openaire.exporter.jdbc.maxIdle      =   20
+openaire.exporter.jdbc.maxRows      =   100
+
+# PROJECTS
+openaire.exporter.project.dspaceHeadTemplate    =   classpath:/eu/dnetlib/openaire/st/projects_dspace_header.st
+openaire.exporter.project.dspaceTemplate        =   classpath:/eu/dnetlib/openaire/st/projects_dspace_project.st
+openaire.exporter.project.dspaceTailTemplate    =   classpath:/eu/dnetlib/openaire/st/projects_dspace_tail.st
+openaire.exporter.project.eprintsTemplate       =   classpath:/eu/dnetlib/openaire/st/projects_eprints.st
+openaire.exporter.project.tsvFields             =   Grant Agreement Number, Project Acronym, Project Title, Call ID, Start Date, End Date, OA Mandate on Publications, OA Mandate on Datasets, Discipline, Organization, Country, Role, Person Name, Person Second Names, Person Email
+openaire.exporter.project.projectsFundingQueryTemplate=classpath:/eu/dnetlib/openaire/sql/projects_fundings.sql.st
+openaire.exporter.project.flushSize             =   1000
+
+# DATSOURCES
+openaire.exporter.datasource.title		    =	Data Sources
+openaire.exporter.datasource.mongoHost      =   localhost
+openaire.exporter.datasource.mongoPort      =   27017
+openaire.exporter.datasource.mongoConnectionsPerHost = 10
+openaire.exporter.datasource.mongoCollectionName = wf_logs
+openaire.exporter.datasource.mongoDbName    =   dnet_logs_prod
+openaire.exporter.datasource.mongoQueryLimit=   100
+openaire.exporter.findSolrIndexUrl          =   /eu/dnetlib/openaire/xquery/findSolrIndexUrl.xquery
+openaire.exporter.findIndexDsInfo           =   /eu/dnetlib/openaire/xquery/findIndexDsInfo.xquery
+openaire.exporter.findObjectStore           =   /eu/dnetlib/openaire/xquery/findObjectStore.xquery
+openaire.exporter.findFunderContexts        =   /eu/dnetlib/openaire/xquery/findFunderContexts.xquery
+openaire.exporter.findCommunityContexts     =   /eu/dnetlib/openaire/xquery/findCommunityContexts.xquery
+openaire.exporter.findContextProfiles       =   /eu/dnetlib/openaire/xquery/findContextProfiles.xquery
+openaire.exporter.findContextProfilesByType =   /eu/dnetlib/openaire/xquery/findContextProfilesByType.xquery
+openaire.exporter.getRepoProfile            =   /eu/dnetlib/openaire/xquery/getRepoProfile.xquery
+
+openaire.exporter.contentLoadQuery          =   { "$and" : [ { "system:profileName" : "Graph construction [PROD]" }, { "system:isCompletedSuccessfully" : "true" }, { "reuseContent" : "false" } ] }
+
+# REST API CONFIGURATION
+openaire.exporter.swaggerDsm.apiTitle          =   OpenAIRE aggregator REST API
+openaire.exporter.swaggerDsm.apiDescription    =   The OpenAIRE data provision REST API allows developers to access the metadata information space of OpenAIRE programmatically.
+openaire.exporter.swaggerDsm.apiLicense        =   LICENSED UNDER GNU AFFERO GENERAL PUBLIC LICENSE.
+openaire.exporter.swaggerDsm.apiLicenseUrl     =   https://www.gnu.org/licenses/agpl-3.0.txt
+openaire.exporter.swaggerDsm.apiContacName     =   D-Net team
+openaire.exporter.swaggerDsm.apiContactUrl     =   http://www.openaire.eu
+openaire.exporter.swaggerDsm.apiContactEmail   =   dnet-team@isti.cnr.it
+
+openaire.exporter.swaggerProjects.apiTitle          =   OpenAIRE projects REST API
+openaire.exporter.swaggerProjects.apiDescription    =   The OpenAIRE projects REST API allows programmatic access to funded research projects metadata.
+openaire.exporter.swaggerProjects.apiLicense        =   ${openaire.exporter.swaggerDsm.apiLicense}
+openaire.exporter.swaggerProjects.apiLicenseUrl     =   ${openaire.exporter.swaggerDsm.apiLicenseUrl}
+openaire.exporter.swaggerProjects.apiContacName     =   ${openaire.exporter.swaggerDsm.apiContacName}
+openaire.exporter.swaggerProjects.apiContactUrl     =   ${openaire.exporter.swaggerDsm.apiContactUrl}
+openaire.exporter.swaggerProjects.apiContactEmail   =   ${openaire.exporter.swaggerDsm.apiContactEmail}
+
+openaire.exporter.swaggerFunders.apiTitle          =   OpenAIRE funders REST API
+openaire.exporter.swaggerFunders.apiDescription    =   The OpenAIRE funders REST API allows programmatic access to the funding agencies metadata in OpenAIRE.
+openaire.exporter.swaggerFunders.apiLicense        =   ${openaire.exporter.swaggerDsm.apiLicense}
+openaire.exporter.swaggerFunders.apiLicenseUrl     =   ${openaire.exporter.swaggerDsm.apiLicenseUrl}
+openaire.exporter.swaggerFunders.apiContacName     =   ${openaire.exporter.swaggerDsm.apiContacName}
+openaire.exporter.swaggerFunders.apiContactUrl     =   ${openaire.exporter.swaggerDsm.apiContactUrl}
+openaire.exporter.swaggerFunders.apiContactEmail   =   ${openaire.exporter.swaggerDsm.apiContactEmail}
+
+openaire.exporter.swaggerCommunities.apiTitle          =   OpenAIRE Communities REST API
+openaire.exporter.swaggerCommunities.apiDescription    =   The OpenAIRE communities REST API allows programmatic access to the communities configurations in OpenAIRE.
+openaire.exporter.swaggerCommunities.apiLicense        =   ${openaire.exporter.swaggerDsm.apiLicense}
+openaire.exporter.swaggerCommunities.apiLicenseUrl     =   ${openaire.exporter.swaggerDsm.apiLicenseUrl}
+openaire.exporter.swaggerCommunities.apiContacName     =   ${openaire.exporter.swaggerDsm.apiContacName}
+openaire.exporter.swaggerCommunities.apiContactUrl     =   ${openaire.exporter.swaggerDsm.apiContactUrl}
+openaire.exporter.swaggerCommunities.apiContactEmail   =   ${openaire.exporter.swaggerDsm.apiContactEmail}
+
+openaire.exporter.swaggerContexts.apiTitle          =   OpenAIRE Contexts REST API
+openaire.exporter.swaggerContexts.apiDescription    =   The OpenAIRE contexts REST API allows programmatic access to the context profiles in OpenAIRE.
+openaire.exporter.swaggerContexts.apiLicense        =   ${openaire.exporter.swaggerDsm.apiLicense}
+openaire.exporter.swaggerContexts.apiLicenseUrl     =   ${openaire.exporter.swaggerDsm.apiLicenseUrl}
+openaire.exporter.swaggerContexts.apiContacName     =   ${openaire.exporter.swaggerDsm.apiContacName}
+openaire.exporter.swaggerContexts.apiContactUrl     =   ${openaire.exporter.swaggerDsm.apiContactUrl}
+openaire.exporter.swaggerContexts.apiContactEmail   =   ${openaire.exporter.swaggerDsm.apiContactEmail}
+
+openaire.exporter.swaggerInfo.apiTitle          =   OpenAIRE Info REST API
+openaire.exporter.swaggerInfo.apiDescription    =   The OpenAIRE Info REST API allows programmatic access to some interesting dates related to the content indexed in OpenAIRE.
+openaire.exporter.swaggerInfo.apiLicense        =   ${openaire.exporter.swaggerDsm.apiLicense}
+openaire.exporter.swaggerInfo.apiLicenseUrl     =   ${openaire.exporter.swaggerDsm.apiLicenseUrl}
+openaire.exporter.swaggerInfo.apiContacName     =   ${openaire.exporter.swaggerDsm.apiContacName}
+openaire.exporter.swaggerInfo.apiContactUrl     =   ${openaire.exporter.swaggerDsm.apiContactUrl}
+openaire.exporter.swaggerInfo.apiContactEmail   =   ${openaire.exporter.swaggerDsm.apiContactEmail}
+
+# VOCABULARIES
+openaire.exporter.vocabularies.baseUrl                      = http://localhost:8980/provision/mvc/vocabularies
+openaire.exporter.vocabularies.countriesEndpoint            = ${openaire.exporter.vocabularies.baseUrl}/dnet:countries.json
+openaire.exporter.vocabularies.datasourceTypologiesEndpoint = ${openaire.exporter.vocabularies.baseUrl}/dnet:datasource_typologies.json

apps/dnet-exporter-api/src/main/resources/sql/community-schema.sql

@@ -0,0 +1,87 @@
+DROP TABLE IF EXISTS community_subs;
+DROP TABLE IF EXISTS community_projects;
+DROP TABLE IF EXISTS community_datasources;
+DROP TABLE IF EXISTS community_support_orgs;
+DROP TABLE IF EXISTS community_orgs;
+DROP TABLE IF EXISTS communities;
+
+CREATE TABLE communities (
+	id                       text PRIMARY KEY,
+	name                     text NOT NULL,
+	shortname                text NOT NULL, -- in the profile is label
+	displayname              text,
+	displayshortname         text,
+	description              text NOT NULL DEFAULT '',
+	status                   text NOT NULL DEFAULT 'hidden', -- all, manager, hidden, members
+	membership               text NOT NULL DEFAULT 'by-invitation', -- open, by-invitation
+	type                     text NOT NULL, -- community, ri
+	claim                    text, -- managers-only, members-only, all
+	subjects                 text[],
+	fos                      text[],
+	sdg                      text[],
+	adv_constraints          jsonb,
+	remove_constraints       jsonb,
+	main_zenodo_community    text,
+	other_zenodo_communities text[],
+	creation_date            timestamp NOT NULL DEFAULT now(),
+	last_update              timestamp NOT NULL DEFAULT now(),
+	logo_url                 text,
+	suggested_acknowledgements text[],
+	plan                     text NOT NULL DEFAULT 'Default'
+);
+
+CREATE TABLE community_projects (
+	community       text NOT NULL REFERENCES communities(id),
+	project_id      text NOT NULL,
+	project_code    text NOT NULL,
+	project_name    text NOT NULL,
+	project_acronym text,
+	project_funder  text NOT NULL,
+	available_since date NOT NULL default now(),
+	PRIMARY KEY (community, project_id)
+);
+
+CREATE TABLE community_datasources (
+	community       text NOT NULL REFERENCES communities(id),
+	ds_id           text NOT NULL,
+	ds_name         text NOT NULL,
+	ds_officialname text NOT NULL,
+	enabled         boolean NOT NULL DEFAULT true
+	constraints     jsonb,
+	deposit         boolean NOT NULL DEFAULT false,
+	message         text,
+	PRIMARY KEY (community, ds_id)
+);
+
+CREATE TABLE community_support_orgs (
+	community     text NOT NULL REFERENCES communities(id),
+	org_name      text NOT NULL,
+	org_url       text NOT NULL,
+	org_logourl   text NOT NULL,
+	PRIMARY KEY (community, org_name)
+);
+
+CREATE TABLE community_orgs (
+	community   text NOT NULL REFERENCES communities(id),
+	org_id      text NOT NULL,
+	PRIMARY KEY (community, org_id)
+);
+
+
+CREATE TABLE community_subs (
+	sub_id      text    NOT NULL PRIMARY KEY,
+	community   text    NOT NULL REFERENCES communities(id),
+	label       text    NOT NULL,
+	category    text    NOT NULL,
+	claim       boolean NOT NULL DEFAULT false,
+	browsable   boolean NOT NULL DEFAULT false,
+	params      jsonb,
+	parent      text REFERENCES community_subs(sub_id) -- NULL for the first level
+);
+
+CREATE INDEX community_projects_community ON community_projects(community);
+CREATE INDEX community_datasources_community ON community_datasources(community);
+CREATE INDEX community_support_orgs_community ON community_support_orgs(community);
+CREATE INDEX community_orgs_community ON community_orgs(community);
+CREATE INDEX community_subs_community ON community_subs(community);
+CREATE INDEX community_subs_parent ON community_subs(parent);

apps/dnet-exporter-api/src/main/resources/sql/export_nwo_projects.sql

@@ -0,0 +1,11 @@
+\copy (
+	SELECT 
+		'netherlands'                         AS community,
+		'nwo_________::'||md5(substr(id, 15)) AS project_id,
+		code                                  AS project_code,
+		title                                 AS project_name,
+		acronym                               AS project_acronym,
+		'NWO'                                 AS project_funder
+	FROM projects
+	WHERE id LIKE 'nwo\_\_\_\_\_\_\_\_\_::%'
+) TO '/tmp/nwo_community_projects.csv' CSV HEADER;

apps/dnet-exporter-api/src/main/resources/sql/funders-schema.sql

@@ -0,0 +1,25 @@
+ALTER TABLE dsm_organizations ADD COLUMN registered_funder boolean;
+
+CREATE VIEW funders_view AS SELECT
+	o.id                AS id, 
+	o.legalshortname    AS legalshortname,
+	o.legalname         AS legalname,
+	o.websiteurl        AS websiteurl, 
+	o.logourl           AS logourl, 
+	o.country           AS country,
+	o.dateofcollection  AS registrationdate,
+	o.registered_funder AS registered,
+	array_agg(DISTINCT s.id||' @=@ '||s.officialname||' @=@ '||s.eosc_datasource_type) AS datasources,
+	array_agg(DISTINCT pids.issuertype||' @=@ '||pids.pid) AS pids
+FROM 
+	dsm_organizations o 
+	JOIN dsm_service_organization so           ON (o.id = so.organization) 
+	JOIN dsm_services s                        ON (so.service = s.id) 
+	JOIN projects p                            ON (p.collectedfrom = s.id)
+	LEFT OUTER JOIN dsm_organizationpids opids ON (o.id = opids.organization)
+	LEFT OUTER JOIN dsm_identities pids        ON (opids.pid = pids.pid)
+GROUP BY o.id;
+
+GRANT ALL ON funders_view TO dnetapi;
+
+

apps/dnet-exporter-api/src/main/resources/sql/import_nwo_projects.sql

@@ -0,0 +1 @@
+\copy community_projects FROM '/tmp/nwo_community_projects.csv' CSV HEADER;

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/community/CommunityApiControllerTest.java

@@ -8,7 +8,7 @@ import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.
 import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
 
 import java.nio.charset.Charset;
-import java.util.Date;
+import java.time.LocalDateTime;
 import java.util.List;
 
 import org.junit.jupiter.api.Test;
@@ -19,12 +19,14 @@ import org.springframework.boot.test.mock.mockito.MockBean;
 import org.springframework.http.MediaType;
 import org.springframework.test.web.servlet.MockMvc;
 
+import eu.dnetlib.openaire.exporter.model.community.CommunitySummary;
+
 /**
  * Created by Alessia Bardi on 2019-04-04.
  *
  * @author Alessia Bardi
  */
-@SpringBootTest
+//@SpringBootTest
 @WebMvcTest(CommunityApiController.class)
 public class CommunityApiControllerTest {
 
@@ -44,7 +46,7 @@ public class CommunityApiControllerTest {
 		final CommunitySummary cs = new CommunitySummary();
 		cs.setDescription("the description");
 		cs.setId("id1");
-		cs.setLastUpdateDate(new Date());
+		cs.setLastUpdateDate(LocalDateTime.now());
 		cs.setName("X");
 		cs.setShortName("x");
 		final List<CommunitySummary> csList = singletonList(cs);

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/community/importer/CommunityImporterServiceTest.java

@@ -0,0 +1,161 @@
+package eu.dnetlib.openaire.community.importer;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+
+import org.apache.commons.io.IOUtils;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.springframework.jdbc.core.JdbcTemplate;
+
+import eu.dnetlib.openaire.community.CommunityService;
+import eu.dnetlib.openaire.community.model.DbOrganization;
+import eu.dnetlib.openaire.community.repository.DbOrganizationRepository;
+import eu.dnetlib.openaire.context.ContextMappingUtils;
+import eu.dnetlib.openaire.exporter.model.community.CommunityContentprovider;
+import eu.dnetlib.openaire.exporter.model.community.CommunityDetails;
+import eu.dnetlib.openaire.exporter.model.community.CommunityOrganization;
+import eu.dnetlib.openaire.exporter.model.community.CommunityProject;
+import eu.dnetlib.openaire.exporter.model.community.SubCommunity;
+import eu.dnetlib.openaire.exporter.model.context.Context;
+
+@ExtendWith(MockitoExtension.class)
+class CommunityImporterServiceTest {
+
+	// Class under test
+	private CommunityImporterService importer;
+
+	@Mock
+	private DbOrganizationRepository dbOrganizationRepository;
+
+	@Mock
+	private CommunityService service;
+
+	@Mock
+	private JdbcTemplate jdbcTemplate;
+
+	@BeforeEach
+	public void setUp() {
+		importer = new CommunityImporterService();
+		importer.setDbOrganizationRepository(dbOrganizationRepository);
+		importer.setService(service);
+		importer.setJdbcTemplate(jdbcTemplate);
+	}
+
+	@Test
+	public void testImportPropagationOrganizationsFromProfile() throws Exception {
+		final String profile = IOUtils.toString(getClass().getResourceAsStream("old_provision_wf.xml"), StandardCharsets.UTF_8.toString());
+		final List<DbOrganization> list = importer.importPropagationOrganizationsFromProfile(profile, true);
+		// list.forEach(System.out::println);
+
+		assertEquals(245, list.size());
+		assertEquals(1, list.stream().filter(o -> "openorgs____::9dd5545aacd3d8019e00c3f837269746".equals(o.getOrgId())).count());
+		assertEquals(2, list.stream().filter(o -> "openorgs____::d11f981828c485cd23d93f7f24f24db1".equals(o.getOrgId())).count());
+		assertEquals(14, list.stream().filter(o -> "beopen".equals(o.getCommunity())).count());
+	}
+
+	@SuppressWarnings("unchecked")
+	@Test
+	public void testImportCommunity() throws Exception {
+		final String profile = IOUtils.toString(getClass().getResourceAsStream("old_community_profile.xml"), StandardCharsets.UTF_8.toString());
+
+		final Queue<Throwable> errors = new LinkedList<>();
+		final Context context = ContextMappingUtils.parseContext(profile, errors);
+		assertTrue(errors.isEmpty());
+
+		Mockito.when(jdbcTemplate.queryForList(Mockito.anyString(), Mockito.any(Class.class), Mockito.anyString())).thenReturn(Arrays.asList("corda_______"));
+		importer.importCommunity(context);
+
+		final ArgumentCaptor<CommunityDetails> detailsCapture = ArgumentCaptor.forClass(CommunityDetails.class);
+		final ArgumentCaptor<CommunityProject> projectsCapture = ArgumentCaptor.forClass(CommunityProject.class);
+		final ArgumentCaptor<CommunityContentprovider> datasourcesCapture = ArgumentCaptor.forClass(CommunityContentprovider.class);
+		final ArgumentCaptor<CommunityOrganization> orgsCapture = ArgumentCaptor.forClass(CommunityOrganization.class);
+		final ArgumentCaptor<SubCommunity> subCommunitiesCapture = ArgumentCaptor.forClass(SubCommunity.class);
+
+		Mockito.verify(service, Mockito.times(1)).saveCommunity(detailsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityProjects(Mockito.anyString(), projectsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityDatasources(Mockito.anyString(), datasourcesCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityOrganizations(Mockito.anyString(), orgsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addSubCommunities(Mockito.anyString(), subCommunitiesCapture.capture());
+
+		final CommunityDetails details = detailsCapture.getValue();
+		assertEquals("egi", details.getId());
+		// System.out.println(details);
+
+		final List<CommunityProject> projects = projectsCapture.getAllValues();
+		assertEquals(83, projects.size());
+		// projects.forEach(System.out::println);
+
+		final List<CommunityContentprovider> datasources = datasourcesCapture.getAllValues();
+		assertEquals(1, datasources.size());
+		// datasources.forEach(System.out::println);
+
+		final List<CommunityOrganization> orgs = orgsCapture.getAllValues();
+		assertEquals(1, orgs.size());
+		// orgs.forEach(System.out::println);
+
+		final List<SubCommunity> subs = subCommunitiesCapture.getAllValues();
+		assertEquals(688, subs.size());
+		// subs.forEach(System.out::println);
+
+	}
+
+	@SuppressWarnings("unchecked")
+	@Test
+	public void testImportCommunityFetFp7() throws Exception {
+		final String profile = IOUtils.toString(getClass().getResourceAsStream("old_community_profile_fet-fp7.xml"), StandardCharsets.UTF_8.toString());
+
+		final Queue<Throwable> errors = new LinkedList<>();
+		final Context context = ContextMappingUtils.parseContext(profile, errors);
+		assertTrue(errors.isEmpty());
+
+		// Mockito.when(jdbcTemplate.queryForList(Mockito.anyString(), Mockito.any(Class.class),
+		// Mockito.anyString())).thenReturn(Arrays.asList("corda_______"));
+		importer.importCommunity(context);
+
+		final ArgumentCaptor<CommunityDetails> detailsCapture = ArgumentCaptor.forClass(CommunityDetails.class);
+		final ArgumentCaptor<CommunityProject> projectsCapture = ArgumentCaptor.forClass(CommunityProject.class);
+		final ArgumentCaptor<CommunityContentprovider> datasourcesCapture = ArgumentCaptor.forClass(CommunityContentprovider.class);
+		final ArgumentCaptor<CommunityOrganization> orgsCapture = ArgumentCaptor.forClass(CommunityOrganization.class);
+		final ArgumentCaptor<SubCommunity> subCommunitiesCapture = ArgumentCaptor.forClass(SubCommunity.class);
+
+		Mockito.verify(service, Mockito.times(1)).saveCommunity(detailsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityProjects(Mockito.anyString(), projectsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityDatasources(Mockito.anyString(), datasourcesCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addCommunityOrganizations(Mockito.anyString(), orgsCapture.capture());
+		Mockito.verify(service, Mockito.times(1)).addSubCommunities(Mockito.anyString(), subCommunitiesCapture.capture());
+
+		final CommunityDetails details = detailsCapture.getValue();
+		assertEquals("fet-fp7", details.getId());
+		// System.out.println(details);
+
+		final List<CommunityProject> projects = projectsCapture.getAllValues();
+		assertEquals(0, projects.size());
+		// projects.forEach(System.out::println);
+
+		final List<CommunityContentprovider> datasources = datasourcesCapture.getAllValues();
+		assertEquals(0, datasources.size());
+		// datasources.forEach(System.out::println);
+
+		final List<CommunityOrganization> orgs = orgsCapture.getAllValues();
+		assertEquals(0, orgs.size());
+		// orgs.forEach(System.out::println);
+
+		final List<SubCommunity> subs = subCommunitiesCapture.getAllValues();
+		assertEquals(151, subs.size());
+		subs.forEach(System.out::println);
+
+	}
+
+}

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/community/selectioncriteria/SelectionCriteriaTest.java

@@ -2,6 +2,10 @@ package eu.dnetlib.openaire.community.selectioncriteria;
 
 import com.google.gson.Gson;
 import eu.dnetlib.data.bulktag.selectioncriteria.Selection;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.Constraint;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.Constraints;
+import eu.dnetlib.openaire.exporter.model.community.selectioncriteria.SelectionCriteria;
+
 import org.junit.jupiter.api.Test;
 
 import org.springframework.beans.factory.annotation.Autowired;

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/context/ContextMappingUtilsTest.java

@@ -13,6 +13,8 @@ import org.junit.jupiter.api.Test;
 import com.google.gson.Gson;
 import com.google.gson.GsonBuilder;
 
+import eu.dnetlib.openaire.exporter.model.context.Context;
+
 /**
  * Created by Alessia Bardi on 2019-04-04.
  *

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/dsm/DatasourceApiControllerTest.java

@@ -21,11 +21,10 @@ import org.springframework.test.web.servlet.MockMvc;
 
 import eu.dnetlib.enabling.datasources.common.DsmException;
 import eu.dnetlib.openaire.dsm.dao.utils.DsmMappingUtils;
-import eu.dnetlib.openaire.dsm.domain.DatasourceDetails;
 import eu.dnetlib.openaire.dsm.domain.db.DatasourceDbEntry;
-import eu.dnetlib.openaire.vocabularies.Country;
+import eu.dnetlib.openaire.exporter.model.dsm.DatasourceDetails;
+import eu.dnetlib.openaire.exporter.model.vocabularies.Country;
 
-@SpringBootTest
 @WebMvcTest(DsmApiController.class)
 public class DatasourceApiControllerTest {
 

apps/dnet-exporter-api/src/test/java/eu/dnetlib/openaire/dsm/dao/utils/DsmMappingUtilsTest.java

@@ -5,8 +5,8 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 
-import eu.dnetlib.openaire.dsm.domain.ApiDetails;
 import eu.dnetlib.openaire.dsm.domain.db.ApiDbEntry;
+import eu.dnetlib.openaire.exporter.model.dsm.ApiDetails;
 
 class DsmMappingUtilsTest {