README.md

# Unified Data Structure db scrips

Contains:
* script to import file data to db
* db queries Python/R &JSON files or embedded in python or R scripts
* sample code to use that

## Install

* Read and apply [first-time git setup instructions](./GIT_SETUP.md).
* Clone git repo: 

  ```bash
  git clone https://jeodpp.jrc.ec.europa.eu/apps/gitlab/use_cases/legent/uds-scripts.git
  ```
* Run this bash command to append scripts-path into $PATH and $PYTHONPATH:

```bash
source ./install.sh
```

## Usage

* CD to a dir where you are going to work,
  inside your home-dir, eg: `/home/{user}/{foo}`
  (this may be a new one)
* --(process db-data)--
  * run scripts & queries to get the data from the db (and possibly create files)
    * BASH-HISTORY is your friend
  * modify script-files
  * (optional) update db
* copy/move specific files for the NextCloud agent to pick them up,
  from:
  
  ```
  /home/{user}/Documents/{foo} --> /home/{user}/{foo}/Documents
                               --> /eos/jeodpp/home/users/{user}/{bar}
                               --> /eos/jeodpp/data/projects/LEGENT/transfer/{baz}
  ```

### Sample code

```python
from db import eea_2020_flattened
from utils import save_to_parquet
from config import Config
import pandas as pd

df = pd.DataFrame.from_records(eea_2020_flattened.find())

...  # process your data

save_to_parquet(df, Config.PUBLIC_SAVE_PATH, 'test.parquet')
save_to_excel(df, Config.PUBLIC_SAVE_PATH, 'test.xlsx')
```

### NextCloud mapping of folders

```
BDAP                                           PC-folder of NextCloud
=========================================      =========================
/home/{user}/Documents                    <--> ~/BDAPCloud/data
/eos/jeodpp/home/users/{user}             <--> ~/BDAPCloud/eos_user_home
/eos/jeodpp/data/projects/LEGENT/transfer <--> ~/BDAPCloud/eos_LEGENT
```