This site uses cookies. Visit our cookies policy page or click the link in any footer for more information and to change your preferences.

Accept all cookies Accept only essential cookies

Code development platform for open source projects from the European Union institutions :large_blue_circle: EU Login authentication by SMS has been phased out. To see alternatives please check here

Skip to content
Snippets Groups Projects
Commit 11dbaf5d authored by Thomas Vliagkoftis's avatar Thomas Vliagkoftis
Browse files

enh: update README.md delete GIT_SETUP.md

parent e95b05e3
Branches
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
GIT_SETUP.md deleted 100644 → 0
+ 0
− 184
View file @ e95b05e3
---
adapted-from: https://app.clickup.com/2189764/v/dc/22ue4-596/22ue4-358
---
# Access GitLab
1\. Environment requirements
============================
You will be using the following programs the jeodpp provides:
* Bash.
* PyCharm.
* Compass the GUI for MongoDB (soon to be available).
2\. Configure authentication
============================
2.1 Access Tokens
-----------------
Create a new access token:
* Log into Gitlab [https://gitlab.jrc.nl/](https://gitlab.jrc.nl/)and click your Profile -> Settings -> Access Tokens
![](https://t2189764.p.clickup-attachments.com/t2189764/934070d9-de92-4f6a-bc29-c979f7cc7909/image.png)
* Create a new access token:
* Choose a self-explanatory name, eg <you-username>@<your-PC-name>
* tick `read_repository` and `write_repository` scopes
* click `Create personal access token` button
* Copy the token into your clipboard - it will be only visible until you close/refresh the page; afterwards you would have to re-create it.
* Try to clone this repo inside your **home dir** (`/home/$USER`):
```bash
cd ~
git clone https://jeodpp.jrc.ec.europa.eu/apps/gitlab/use_cases/legent/uds-scripts.git/
```
**DO WE NEED THIS LINE?** OR add a new remote into your _existing_ local git-repo and then try to fetch:
2.2. SSH keys
-------------
SSH keys are (probably?) needed if you access gitlab through `git://...` url-schemes.
**Steps**
1. Generate SSH key
Gitlab [https://gitlab.jrc.nl/](https://gitlab.jrc.nl/) ->Profile -> Settings -> Add SSH Key
![](https://t2189764.p.clickup-attachments.com/t2189764/35cc5eb8-1514-4331-8b34-e594a2e8220f/image.png)
Open Command line E:\\vecto\\clone\\vecto>ssh-keygen -t rsa -b 2048 -C "mail"
Copy id\_rsa.pub to the keys from Gitlab settings (.ssh/id\_rsa.pub)
Alternative, use Access Token. Generate token and copy. The clone will use https instead of ssh.
2\. Go to Gitlab petten Vecto Project
![](https://t2189764.p.clickup-attachments.com/t2189764/eae70443-3232-4fb9-8072-54fb6bf928a1/image.png)
2\. Fork the project into a personal repo
=========================================
For the git workflow selected by the team, each developer works on an independent fork of the project visible to all the team members. ([Forking workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow#:~:text=Forking%20is%20a%20git%20clone,org%2FuserA%2Fopen%2Dproject))
To create this fork click on the fork button at the top of the project page:
![](https://t2189764.p.clickup-attachments.com/t2189764/38c1a98c-9470-43f3-917c-7e0e1e89d702/image.png)
_Note:_ As a new team member of the team you might encounter an issue indicating that the limit of forks for this project has been reached:
![](https://t2189764.p.clickup-attachments.com/t2189764/0b406e7c-ab34-4f6c-9ccf-0be1d6160c49/image.png)
In that case contact the system administrator and ask them to change the project limit for your account.
After adding the name information for the fork, it is necessary to give permissions to the rest of the team to make it visible for them. For this, go to the url:
[https://gitlab.jrc.nl/YOURUSERNAME/vecto/-/project\_members](https://gitlab.jrc.nl/munozme/vecto/-/project_members)
There select the tab _Invite group_ and add VECTO.
![](https://t2189764.p.clickup-attachments.com/t2189764/c54c2430-63b3-47ef-b237-e31f12212e6d/image.png)
3\. Clone the repo
==================
In case you don't have a local repo open the command line and type the following command, make sure to change the url path to include your username:
![](https://t2189764.p.clickup-attachments.com/t2189764/a4091950-bb6b-4bf9-a83f-a0335d793928/image.png)
In case you already have a local repo (for example, from github) you need to add a new remote and fetch it:
`git remote add {username} https://jeodpp.jrc.ec.europa.eu/apps/gitlab/{username}/uds-scripts`
`git fetch {username}`
4\. Configure your notification email
=====================================
The primary email assigned to your account is your EU email. This email is used for authentication and notifications. To view your gitlab notifications you need to login to your email and check the new notifications, which means you won't be able to see as soon as you get them unless you're logged into your EU account.
An alternative to see your notifications as soon as they arrive is changing your notification email to another email account.
To do this first access [https://gitlab.jrc.nl/-/profile/emails](https://gitlab.jrc.nl/-/profile/emails) and add a secondary email address to your account. You'll receive an email from gitlab asking you to verify your account. Then go to [https://gitlab.jrc.nl/-/profile/notifications](https://gitlab.jrc.nl/-/profile/notifications) and select the email address you just added as a notification address.
5\. Rename your remote
======================
The default name for a remote is origin, and given that we work all in forks of the same project but can also make changes to someone else's fork, we decided to rename the each remote. The new name corresponds to an identifier of the server and of the username of choice as long as is brief. It looks like "ptuser". The idea is to facilitate differentiating to which remote a change is being pushed and to be able to replicate a command to an specific remote without having to change the command. To rename use the command:
```plain
git remote rename origin ptuser
```
pro-tip: for _remoting_ commands, you may use directly the _url-of-a-repo_ in place of a _remote-name, eg:_
```plain
git push https://github.com/some/repo.git some-branch
```
\ No newline at end of file
README.md
+ 88
− 30
View file @ 11dbaf5d
......@@ -7,37 +7,93 @@ preloaded in *eos-dirs* (read section below about Nextcloud mapping).
## 0. Contents
* `build_eea.py` - script to import EEA's files into mongo db
* Scripts to import various datasets into mongo db
* `example.py` - sample script to query db and extract docs ("rows") to files
* `src/uds4jrc/` - python project to support the scripts, acting as their library
* data from those C4 projects & activities:
* EEA - vehicle registrations in EU, reported by MSs & packaged by EEA (raw data)
* Fiat500x - campaign with OBD data from JRC "amateur" drivers - TODO
* Fiat500x - campaign with OBD data from JRC "amateur" drivers
* RealWorld (Travelcard, Geco Air, Spritmonitor)
* ATCT
* DICE - TODO
* commercial vehicle specs - TODO
* ...
## 1. Installation
## 1. Environment requirements
> **TIP:** Prefer _Chrome browser_ when logging into BDAP.
>
> _Chrome_ is a much better fit for BDAP's JEODesk service;
> _Firefox_ does not send any of its default keyboard shortcuts to the _Gucamole_-based
> web-page (ie to the terminal or the IDE you run in BDAP).
> For instance, **[Ctrl+U]/[Ctrl+U]** in _firefox_ do not kill chars left/right of the cursor,
> For instance, **[Ctrl+U]/[Ctrl+K]** in _firefox_ do not kill chars left/right of the cursor,
> but _"Show page-source"_ & _"Search in address-bar"_ instead, respectively.
1. Read and apply [first-time `git@BDAP` setup instructions](./GIT_SETUP.md).
You will be using the following programs the jeodpp provides:
* Bash.
* PyCharm.
* Compass the GUI for MongoDB.
## 2. Configure authentication
### 2.1 Access Tokens
Create a new access token:
* Log into Gitlab [https://jeodpp.jrc.ec.europa.eu/apps/gitlab/](https://jeodpp.jrc.ec.europa.eu/apps/gitlab) and click your Profile -> Settings -> Access Tokens
![](doc/gitlab_access_tokens.png)
* Create a new access token:
* Choose a self-explanatory name, eg <you-username>@<your-PC-name>
* tick `read_repository` and `write_repository` scopes
* click `Create personal access token` button
* Copy the token into your clipboard - it will be only visible until you close/refresh the page; afterwards you would have to re-create it.
* Create a file in you jeodpp home directory and save you token there for easy access
## 3. Fork the project into a personal repo
For the git workflow selected by the team, each developer works on an independent fork of the project visible to all the team members. ([Forking workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow#:~:text=Forking%20is%20a%20git%20clone,org%2FuserA%2Fopen%2Dproject))
To create this fork click on the fork button at the top right of the project page and then:
* Select your name from amongst the available users to compliment the *Project URL* section
* Select **Internal** for the *Visibility level*
* Click *Fork project*
## 4. Additional GitLab configuration
### 4.1 Configure your notification email
2. Navigate inside your previously cloned **repo folder** (`/home/$USER/uds-scripts`):
The primary email assigned to your account is your EU email. This email is used for authentication and notifications. To view your gitlab notifications you need to login to your email and check the new notifications, which means you won't be able to see as soon as you get them unless you're logged into your EU account.
An alternative to see your notifications as soon as they arrive is changing your notification email to another email account.
To do this first access [https://jeodpp.jrc.ec.europa.eu/apps/gitlab/-/profile/emails](https://jeodpp.jrc.ec.europa.eu/apps/gitlab/-/profile/emails) and add a secondary email address to your account. You'll receive an email from gitlab asking you to verify your account. Then go to [https://jeodpp.jrc.ec.europa.eu/apps/gitlab/-/profile/notifications](https://jeodpp.jrc.ec.europa.eu/apps/gitlab/-/profile/notifications) and select the email address you just added as a notification address.
## 5. Installation
1. Try to clone this repo inside your **home dir** (`/home/$USER`):
```bash
cd ~
git clone https://jeodpp.jrc.ec.europa.eu/apps/gitlab/use_cases/legent/uds-scripts.git/
```
2. Navigate inside your cloned **repo folder** (`/home/$USER/uds-scripts`):
```bash
cd ~/uds-scripts
```
3. Run the `setup-account.sh` script to setup your BDAP linux account for the first time,
3. Add your fork remote and fetch it:
```bash
git remote add {username} https://jeodpp.jrc.ec.europa.eu/apps/gitlab/{username}/uds-scripts
git fetch {username}
```
4. Run the `setup-account.sh` script to setup your BDAP linux account for the first time,
and create a new conda-environment to install the project:
```bash
......@@ -45,17 +101,17 @@ preloaded in *eos-dirs* (read section below about Nextcloud mapping).
```
> **NOTE:** the command will modify your BDAP's rc-files to offer:
> - searching terminal history with UP arrow,
> - autocompletion for `git` & `conda` commands,
> - bash & ipython aliases & variables to *eos-dirs*,
> - prepare your `git` identity, for when committing.
- searching terminal history with UP arrow,
- autocompletion for `git` & `conda` commands,
- bash & ipython aliases & variables to *eos-dirs*,
- prepare your `git` identity, for when committing.
>
> It's better that you logout and re-login, to pick up the changes,
> but this is not a hard requirement for the next steps.
>
> If everything went as planned, the 2nd command is just `conda activate ./env`
4. Activate the created *conda-env*:
5. Activate the created *conda-env*:
```bash
conda activate ./env
......@@ -63,10 +119,10 @@ preloaded in *eos-dirs* (read section below about Nextcloud mapping).
> **TIP:** you would have to re-issue this command every time you start a new login session or a new bash shell is launched.
5. Create a new file `mysecrets.py` inside the root dir of the cloned project,
6. Create a new file `mysecrets.py` inside the root dir of the cloned project,
and paste inside the db credentials provided by your db administrator.
6. Setup the *conda-env* as your IDE's Python interpreter:
7. Setup the *conda-env* as your IDE's Python interpreter:
* **PyCharm:** File --> Settings --> Project --> Python Interpreter
--> Press gear symbol and select “Add” --> Conda Environment
......@@ -76,14 +132,14 @@ preloaded in *eos-dirs* (read section below about Nextcloud mapping).
"Python: select interpreter" it from the drop-down list,
then navigate to path of the *conda-env*.
7. (optional) install and configure **nextcloud-agent** on your PC,
8. (optional) install and configure **nextcloud-agent** on your PC,
to sync local files with BDAP, like "dropbox" does (read section below).
* For more info on managing Conda environments click [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html).
* For the bdap pycharm/conda documentation click here [here](https://jeodpp.jrc.ec.europa.eu/apps/gitlab/for-everyone/documentation/-/wikis/howto/jeodesk/Set-up-pycharm-with-conda-virtual-environment)
## 1.1. Nextcloud mapping of BDAP folders with your PC
### 5.1. Nextcloud mapping of BDAP folders with your PC
The [*nextcloud* BDAP service](https://jeodpp.jrc.ec.europa.eu/apps/gitlab/for-everyone/documentation/-/wikis/Jeodpp_services/Document-Sharing)
is the only gateway of big data files into and outof BDAP
......@@ -105,11 +161,11 @@ public-eos: /eos/jeodpp/data/projects/LEGENT/transfer <--> eos_LEGENT
> **NOTE:** by default, BDAP maps the 2nd "personal-eos" folder directly to `{user}` dir,
> thus leaving *no space for your experiments* there, beyond nextcloud's mappings;
> and this means wasting network bandwidth & storage on your PC while experimenting.
> >
>
> Please reconfigure *nextcloud-agent* so as to replicate the setup of the "public-eos" dir,
> by creating a `transfer` subdir and limitting nextcloud to sync only that one.
Work with the above dirs is facilitated by the definitons of these Python and
Work with the above dirs is facilitated by the definitions of these Python and
ENVIRONMENT (after installation) variables:
```bash
......@@ -119,7 +175,7 @@ export PEOS="/eos/jeodpp/data/projects/LEGENT"
export PTRANS="/eos/jeodpp/data/projects/LEGENT/transfer"
```
## 2. Usage
## 6. Usage
* activate you conda-env:
......@@ -143,7 +199,7 @@ export PTRANS="/eos/jeodpp/data/projects/LEGENT/transfer"
* copy/move specific files for/to the folders synced by the *nextcloud-agent*,
* or update the "public-eos" dir `$PEOS` (if you're certain about the results).
### 2.1. Sample code
### 6.1. Sample code
```python
from src.uds4jrc.db import eea_2020_flattened
......@@ -166,13 +222,15 @@ save_to_excel(df, Config.UEOS, 'test.xlsx')
> Maybe it's not the best idea to use out bdap's home-ds for experimenting with big-data:
> https://jeodpp.jrc.ec.europa.eu/apps/gitlab/for-everyone/documentation/-/wikis/howto/data/Access-and-transfer-files#storage-places-not-to-be-used-for-data-storage
## 3. Data
## 7. Data
| Name | Type | Path |
|:-------------|:---------:|:-------------------------------------------------------:|
| EEA | raw data | /eos/jeodpp/data/projects/LEGENT/eea |
| f500x | processed | /eos/jeodpp/data/projects/LEGENT/fiat500 |
| Geco Air | raw data | /eos/jeodpp/data/projects/LEGENT/realworld/geco |
| Travelcard | raw data | /eos/jeodpp/data/projects/LEGENT/realworld/travelcard |
| Spritmonitor | raw data | /eos/jeodpp/data/projects/LEGENT/realworld/spritmonitor |
| ATCT | raw data | /eos/jeodpp/data/projects/LEGENT/atct |
\ No newline at end of file
|:-------------|:---------:|:----------------------------------------------------------------:|
| EEA | raw data | /eos/jeodpp/data/projects/LEGENT/internal/eea |
| f500x | processed | /eos/jeodpp/data/projects/LEGENT/internal/fiat500 |
| Geco Air | raw data | /eos/jeodpp/data/projects/LEGENT/internal/realworld/geco |
| Travelcard | raw data | /eos/jeodpp/data/projects/LEGENT/internal/realworld/travelcard |
| Spritmonitor | raw data | /eos/jeodpp/data/projects/LEGENT/internal/realworld/spritmonitor |
| ATCT | raw data | /eos/jeodpp/data/projects/LEGENT/internal/atct |
| CoP | raw data | /eos/jeodpp/data/projects/LEGENT/internal/cop |
| JRCMatics | raw data | /eos/jeodpp/data/projects/LEGENT/internal/jrcmatics |
\ No newline at end of file
doc/gitlab_access_tokens.png 0 → 100644
+ 0
− 0
View file @ 11dbaf5d
doc/gitlab_access_tokens.png

106 KiB

0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment

For further information about the platform and support, please see https://code.europa.eu/info/about