Unified Data Structure db scrips

Contains:

• script to import file data to db
• db queries Python/R &JSON files or embedded in python or R scripts
• sample code to use that

Install

• Read and apply first-time git setup instructions.

• Clone git repo:

  git clone https://jeodpp.jrc.ec.europa.eu/apps/gitlab/use_cases/legent/uds-scripts.git

• Run this bash command to append scripts-path into $PATH and $PYTHONPATH:

source ./install.sh

Usage

• CD to a dir where you are going to work, inside your home-dir, eg: /home/{user}/{foo} (this may be a new one)

• --(process db-data)--

  • run scripts & queries to get the data from the db (and possibly create files)
    • BASH-HISTORY is your friend
  • modify script-files
  • (optional) update db

• copy/move specific files for the NextCloud agent to pick them up, from:

  /home/{user}/Documents/{foo} --> /home/{user}/{foo}/Documents
                               --> /eos/jeodpp/home/users/{user}/{bar}
                               --> /eos/jeodpp/data/projects/LEGENT/transfer/{baz}

Sample code

from db import eea_2020_flattened
from utils import save_to_parquet
from config import Config
import pandas as pd

df = pd.DataFrame.from_records(eea_2020_flattened.find())

...  # process your data

save_to_parquet(df, Config.PUBLIC_SAVE_PATH, 'test.parquet')
save_to_excel(df, Config.PUBLIC_SAVE_PATH, 'test.xlsx')

NextCloud mapping of folders

BDAP                                           PC-folder of NextCloud
=========================================      =========================
/home/{user}/Documents                    <--> ~/BDAPCloud/data
/eos/jeodpp/home/users/{user}             <--> ~/BDAPCloud/eos_user_home
/eos/jeodpp/data/projects/LEGENT/transfer <--> ~/BDAPCloud/eos_LEGENT