====== DSE / NETVIBES Cloud Connector ======

===== 'er-crawler-manager-cli' script Options =====

<code>
Usage: com.ds.apollo.er.crawler.manager.cli.ManagerCliMain [options] [command]
      [command options]
  Options:
    --help

  Commands:
    init      Initialize managed crawl operations to the cloud tenant
      Usage: init

    destroy      Stop managed crawl operations to the cloud tenant and delete
            the existing index unit
      Usage: destroy

    recrawl      Launch a managed re-crawl to the cloud tenant
      Usage: recrawl

    config      List configurations
      Usage: config [options]
        Options:
          --list-crawlable-datasources
            List crawlable data sources configurations
            Default: false
          --list-crawled-datasources
            List crawled data sources configurations
            Default: false

    crawl      Start an unmanaged crawl to the cloud tenant
      Usage: crawl [options]
        Options:
        * --strategy
            Crawl strategy
          --monitored
            True if the crawl should be monitored (true by default)
            Default: true

    test-cloud-connection      Test connection to the cloud tenant services
      Usage: test-cloud-connection
 </code>


===== Checking initial crawl errors =====

Initial crawl job generates a dataset to disk in $CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/er-crawler/storage/datasets/default, containing all successfully crawled document identifiers as well as all errors that happened during that crawl. Objective of this dataset, is to be able to analyze & replay the crawl to recover errors that happened.

To generate an HTML view of this dataset, you can use the provided command line interface: er-crawler-dataset-cli.

  * Linux
<code>$CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/code/command/er-crawler-dataset-cli.sh -D $CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/er-crawler/storage/datasets/default generate-html -o $YOUR_OUTPUT_DIR</code>

  * Windows
<code>%CLOUD_CONNECTOR_INSTALL_DIR%\%ARCH%\code\command\er-crawler-dataset-cli.bat -D %CLOUD_CONNECTOR_INSTALL_DIR%\%ARCH%\er-crawler\storage\datasets\default generate-html -o %YOUR_OUTPUT_DIR%</code>



===== Links =====
  * [[https://help.3ds.com/2025x/english/DSDoc/EXADfsUserMap/exadfs-m-indexing-onpremise-data-sb.htm?contextscope=cloud&id=c858c5ede1374cc89e191130ac3b3db2|DS Doc - Indexing 3DSpace On-premises Data on the Cloud]]
  * [[https://gitlab.paris.exalead.com/apollo/er-crawler/-/wikis/Cloud-Supplemental|Wiki R&D - Cloud Supplemental]]
  * [[https://gitlab.paris.exalead.com/apollo/er-crawler/-/wikis/Processing-Configuration|Wiki R&D - Cloud Supplemental - Filtering data]]
  * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:34OA6fJ5QwOxg9SBzA9l0Q|EDF Road to Cloud - NETVIBES OnPremise Cloud Connector]]
  * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:IHOjbqGPRKKwDT1N52GpeA|EDF Road to Cloud - Administration OP - RAZ & Indexation Full Cloud Connector]]
  * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:TEVNkZegQTeaF_PMSN9Xjw|EDF Road to Cloud - (Internal) Unboxing DATA SCIENCE R2023X/R2024x]]


===== Installation on vdemopro =====

==== Pré-Requis ====

En tant que l'utilisateur 'e4apop', dans le répertoire /home/data/RTV/DEMOR424/, il faut :
  * créer le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
  * copier le tar du connecteur et le décompresser dans le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
  * y copier le tar de tomee et le décompresser dans le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
==== Exécution du script d'installation ====

Il faut lancer le script : ./NETVIBES_CloudConnector.Linux64/1/StartTUI.sh
<code>
[e4apop@vdemopro083abv netvibesCloudConnector]$ ./NETVIBES_CloudConnector.Linux64/1/StartTUI.sh
CurrentMediaDir initial="./NETVIBES_CloudConnector.Linux64/1"
CurrentMediaDir="/home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1"
Current operating system: "Linux"
/usr/bin/lsb_release
DSY_OS_Release="RedHatEnterprise"
DSY_OS="linux_a64"
DSY_LIBPATH_VARNAME="LD_LIBRARY_PATH"
Check Prerequisites for Linux
RedHatEnterprise 8.4
Check Prerequisites for Linux OK.
 INFO: /home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1/inst/linux_a64/code/bin/DSYInsAppliTUI
 INFO: Installer found in current media: "/home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1/inst/linux_a64/code/bin/DSYInsAppliTUI".





________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Welcome to the 3DEXPERIENCE R2022x HotFix 6 installation program.

Please refer to the documentation for the Legal Notices.
Input !q and hit Enter to quit. Just press Enter to continue.

Enter selection (default: Next):

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Choose the installation directory to update or an empty directory to do a new installation:
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [/usr/DassaultSystemes/R2022x/NETVIBES_CloudConnector]: /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Apache TomEE+ installation path
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default []: /home/data/RTV/DEMOR424/netvibesCloudConnector/apache-tomee-plus-8.0.15

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

The embedded Apache TomEE+ will be installed for Testing or Demonstration purposes (not for Production).

Apache TomEE+ connection port
Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.

Default [8080]: 8080

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Choose the server installation directory.
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [/usr/DassaultSystemes/R2022x/3DSpace]: /home/data/RTV/DEMOR424/server

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the 64-bit Java Development Kit (JDK) 11 path.
The minimum required Java version is Oracle JDK 11.0.8 / AdoptOpenJDK-OpenJ9 JDK 11.0.8 (or its IBM Semeru successor) / Oracle JDK 17.0.3 / IBM Semeru JDK 17.0.3, but it is recommended to install the latest Update of Java 11.

To download and install the JDK 11, go to https://www.oracle.com/java/technologies/downloads/#java11 or https://developer.ibm.com/languages/java/semeru-runtimes/downloads/.
It is strongly recommended to use a Java installation path independent from the Java level.

After the Java installation completes, select '< Back' then 'Next >' again to auto-detect the Java path.

Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.

Default []: /home/data/RTV/DEMOR424/jdk

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the username of the crawl user
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [creator]: creator

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the password for the crawl user
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [****]: DemoCenter01


________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Do you want to require users to have a valid CAS session ?
Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.

1 [X] No
Enter selection:


________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the URL of the on cloud 3DRegistry
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [https://GEO-registry.3dx-staging.3ds.com]: https://eu1-registry.3dexperience.3ds.com


________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the name of the cloud tenant
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default []: r1132101374770



________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the on cloud Workspace ID
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [dw-prj-111111-ws]: ffd38927-6958-4bc5-bc85-b7799e71c2dc


________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the Agent ID of an authorized CLM agent to call SGI on the cloud
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default []: fa9f240d-9f14-400a-9d33-da5bf053936b


________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

Enter the password for CLM agent
Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.

Default [****]: B<n(QU+S+z)"cbM"X[%v3CN}

________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

The installation program has enough information to start.
Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to start installation.

Files will be installed in the following directory:
  /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector
Apache TomEE+ installation path: /home/data/RTV/DEMOR424/netvibesCloudConnector/apache-tomee-plus-8.0.15
The embedded Apache TomEE+ will be installed for Testing or Demonstration purposes (not for Production).
  Apache TomEE+ connection port: 8080
The server installation directory: /home/data/RTV/DEMOR424/server
Java Development Kit (JDK) path: /home/data/RTV/DEMOR424/jdk
Crawl username: 'creator'
CAS session validation: 'no'
Cloud Registry URL: 'https://eu1-registry.3dexperience.3ds.com'
Cloud Tenant Name: 'r1132101374770'
Cloud Workspace ID: 'ffd38927-6958-4bc5-bc85-b7799e71c2dc'
Cloud CLM Agent ID: 'fa9f240d-9f14-400a-9d33-da5bf053936b'

Space available: 64.56 GB
Space required: 41.18 MB

Enter selection (default: Install):
Logs are located in "/home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector/InstallData/log/20250902T103230+0200_CODE_linux_a64_NETVIBES_CloudConnector.media"
Copying files...
Installation in progress...
Installing files...
CODE\linux_a64\EXAApoERCrawlerInsCrawlPwCfg (161B)
CODE\linux_a64\EXAApoERCrawlerPreq (63B)
CODE\linux_a64\EXAApoERCrawlerCheckers (5KB)
CODE\linux_a64\EXAApoERCrawlerCommons (103KB)
CODE\linux_a64\EXAApoERCrawlerService (1MB)
CODE\linux_a64\VENSLF4J (787KB)
CODE\linux_a64\RegistryClient (41KB)
CODE\linux_a64\iamCASClient (215KB)
CODE\linux_a64\ENOJavaKernel (2MB)
CODE\linux_a64\ENOFCSJavaClient (441KB)
CODE\linux_a64\VENFasterxmlJackson (4MB)
CODE\linux_a64\VENGoogleGuava (2MB)
CODE\linux_a64\VENApacheCommonsIO (489KB)
CODE\linux_a64\VENApacheCommonsCodec (327KB)
CODE\linux_a64\VENApacheCommonsHttpClient (1MB)
CODE\linux_a64\VENGoogleProtoBuf (1MB)
CODE\linux_a64\VENGson (252KB)
CODE\linux_a64\VENYammerMetrics (83KB)
CODE\linux_a64\EXASupplementalPreq (63B)
CODE\linux_a64\EXASupplementalCommons (3MB)
CODE\linux_a64\EXAApoERCrawlerPreq4OP (14MB)
CODE\linux_a64\EXAApoERCrawlerDataSetCLI (44KB)
CODE\linux_a64\EXAApoERCrawlerManager (183KB)
CODE\linux_a64\EXAApoERCrawlerManagerCLI (76KB)
CODE\linux_a64\EXAApoERCrawlerFileSyncSvc (92KB)
CODE\linux_a64\EXAApoERCrawlerManagerWebapp (6KB)
CODE\linux_a64\EXAApoERCrawlerServiceWebapp (11KB)
CODE\linux_a64\EXAApoERCrawlerInstaller (88KB)
CODE\linux_a64\ERConnectorCommon (239KB)
CODE\linux_a64\ERAgent (1MB)
CODE\linux_a64\VENJavaEE_API (4MB)
CODE\linux_a64\VENApacheJohnzon (253KB)
CODE\linux_a64\VENApacheCommonsLang (688KB)
CODE\linux_a64\NETECOP_TP.prd (6KB)
Installation in progress...
Setting up properties file...
Customizing file...
Executing application...
Generating er-crawler-manager-cli classpath...
Generating er-crawler-dataset-cli classpath...
Generating er-crawler-service-webapp.war...
Generating er-crawler-manager-webapp.war...
Customizing file...
Executing application...
Installing er-crawler...

______________________________________

   Success
______________________________________

Installed er-crawler, you can now start your application server
Once your application server is up, you can initialize the crawl of your tenant by running the following command:
        '/home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector/linux_a64/code/command/er-crawler-manager-cli.sh init'

Actions:
 [1] Ok
______________________________________

Please choose an action: 1
Setting up properties file...
Installation in progress...





________________________________________________________________________________

   3DEXPERIENCE R2022x HotFix 6
________________________________________________________________________________

The installation program has finished installing 3DEXPERIENCE R2022x HotFix 6 on your computer, in /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector.

Just press Enter to exit installation.

Installed Components:
  NETVIBES ER Crawler On Premise TP


Enter selection (default: Close):

</code>

==== Modifications de la config Apache ====
En tant que 'root', aller dans le répertoire '/usr/local/reverseproxy/' et modifier le fichier 'conf/httpd.conf
' en ajoutant, avant le tag </VirtualHost> les lignes suivantes :
<code>
  # Analytics Applications
  ProxyPass        /connector http://vdemoproXXXdsy.extranet.3ds.com:8080
  ProxyPassReverse /connector http://vdemoproXXXdsy.extranet.3ds.com:8080

</code>

Redémarrer le proxy avec une des commandes suivantes
<code>
bin/apachectl restart
</code>
ou
<code>
/usr/local/reverseproxy/bin/httpd -k restart
</code>