Commit 8cdd0218 authored by Christine Plumejeaud's avatar Christine Plumejeaud
Browse files

First code of api, without the postgres backend for the moment. It will come...

First code of api, without the postgres backend for the moment. It will come soon. Many ideas to test to enhance performance.
parent 5ebc9462
- [1. Portic data API : get dynamic data from database](#1-portic-data-api--get-dynamic-data-from-database)
- [1.1. Installation](#11-installation)
- [1.1.1. Running without Apache](#111-running-without-apache)
- [1.1.2. Running with Apache](#112-running-with-apache)
- [1.1.3. Debug things](#113-debug-things)
- [1.2. List of possible requests](#12-list-of-possible-requests)
- [1.2.1. data.portic.fr/api/travels?](#121-dataporticfrapitravels)
- [1.2.2. data.portic.fr/api/pointcalls?](#122-dataporticfrapipointcalls)
- [1.2.3. data.portic.fr/api/details/departures?lon=lat=radius=](#123-dataporticfrapidetailsdepartureslonlatradius)
- [1.2.4. data.portic.fr/api/agg/destinations?lon=lat=radius=](#124-dataporticfrapiaggdestinationslonlatradius)
- [1.2.5. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&both-to=true](#125-dataporticfrapitravelslinktoport%22uhgsid-du-port%22both-totrue)
- [1.2.6. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&degree=0&both-to=true](#126-dataporticfrapitravelslinktoport%22uhgsid-du-port%22degree0both-totrue)
# 1. Portic data API : get dynamic data from database
Web API for data above the postgres database developped with Flask
![](temps_telechargement.png)
## 1.1. Installation
Neep Python 3 to be installed
Clone or download this git repo, then
pip3 install -r requirements.txt
### 1.1.1. Running without Apache
python apidata.py
(or create a virtualenv, then activate it and pip install -r requirements && python apidata.py)
Backend is running and serve the data on http://localhost:5004 (change the PORT variable in config.py)
### 1.1.2. Running with Apache
todo
### 1.1.3. Debug things
C:\Python37\lib\site-packages\flask\app.py:2446: DtypeWarning: Columns (31) have mixed types.Specify dtype option on import or set low_memory=False.
## 1.2. List of possible requests
### 1.2.1. data.portic.fr/api/travels?
-- TOUT en json
### 1.2.2. data.portic.fr/api/pointcalls?
-- TOUT en json
### 1.2.3. data.portic.fr/api/details/departures?lon=lat=radius=
Les travels seulement au départ dans un rayon de radium km autour du point long/lat
Exemple
`http://127.0.0.1:5004/api/agg/departures/?lat=46&lon=-1&radius=100`
```JS
[{"departure":"La Rochelle","count":2631},{"departure":"Marennes","count":1996},{"departure":"Rochefort","count":894},{"departure":"Aligre de Marans","count":705},{"departure":"Saint Martin de R\u00e9","count":673},{"departure":"Ol\u00e9ron","count":600},{"departure":"Rib\u00e9rou Saujon","count":595},{"departure":"Ars en R\u00e9","count":460},{"departure":"La Tremblade","count":320},{"departure":"La Flotte en R\u00e9","count":289},{"departure":"Moricq","count":230},{"departure":"Aligre","count":155},{"departure":"Le Chateau d' Ol\u00e9ron","count":131},{"departure":"La Perrotine","count":121},{"departure":"Oleron","count":115},{"departure":"Saint Denis d' Ol\u00e9ron","count":102},{"departure":"Saint Savinien","count":100},{"departure":"Chaillevette","count":92},{"departure":"Marans","count":82},{"departure":"Olleron","count":79},{"departure":"Saint Michel en l' Herm","count":77},{"departure":"Ile de R\u00e9","count":76},{"departure":"Soubise","count":66},{"departure":"Esnandes","count":62},{"departure":"La Tranche sur Mer","count":56},{"departure":"Seudre","count":54},{"departure":"\u00eele de R\u00e9","count":38},{"departure":"Royan","count":31},{"departure":"Talmont Talmont Saint Hilaire","count":31},{"departure":"\u00cele d' Ol\u00e9ron","count":26},{"departure":"Ile d' Ol\u00e9ron","count":26},{"departure":"Port d' Envaux","count":22},{"departure":"Brouage","count":22},{"departure":"\u00cele de R\u00e9","count":20},{"departure":"Mortagne","count":17},{"departure":"Le Plomb","count":16},{"departure":"Saint Martin de Rh\u00e9","count":12},{"departure":"Loix en R\u00e9","count":12},{"departure":"L' Houm\u00e9e","count":12},{"departure":"la Rochelle","count":11},{"departure":"Taillebourg","count":10},{"departure":"dans le Pertuis","count":10},{"departure":"Dercie","count":8},{"departure":"Seudres","count":8},{"departure":"Saintes","count":8},{"departure":"Martroux","count":8},{"departure":"Bourcefranc","count":6},{"departure":"La Tranche sur mer","count":6},{"departure":"Lu\u00e7on","count":6},{"departure":"La Pr\u00e9e","count":6},{"departure":"Champagn\u00e9 sur Marais","count":5},{"departure":"Fouras","count":4},{"departure":"d' Aligre","count":4},{"departure":"Saint Martin","count":4},{"departure":"La tremblade","count":4},{"departure":"La Perotine","count":4},{"departure":"La Flotte","count":4},{"departure":"Martrou","count":4},{"departure":"la Tremblade","count":4},{"departure":"Marenne","count":4},{"departure":"Courreaux d' Ol\u00e9ron","count":4},{"departure":"Ars En R\u00e9","count":3},{"departure":"la Perrotine","count":3},{"departure":"Ars","count":3},{"departure":"Auvaux Port d' Envaux","count":2},{"departure":"\u00cele d' Oleron","count":2},{"departure":"Saint Michel en Poitou Saint Michel en l' Herm","count":2},{"departure":"Courraux d' Ol\u00e9ron","count":2},{"departure":"R\u00e9","count":2},{"departure":"La Seudre","count":2},{"departure":"Saint Trojan en Ol\u00e9ron","count":2},{"departure":"Rib\u00e9rou","count":2},{"departure":"La B\u00e9chel La B\u00e9chade","count":2},{"departure":"La Tranche","count":2},{"departure":"Aiguillon","count":2},{"departure":"dans Le Pertuis","count":2},{"departure":"Oll\u00e9ron","count":2},{"departure":"Rade de La Pallice","count":2},{"departure":"Coureaux de R\u00e9","count":2},{"departure":"Daligre","count":2},{"departure":"Saint Martin en R\u00e9","count":2},{"departure":"Ile de Re","count":2},{"departure":"L' Aiguillon","count":2},{"departure":"Saint Denis \u00cele d' Ol\u00e9ron","count":2},{"departure":"Sudre","count":2},{"departure":"Le Pr\u00e9e","count":2},{"departure":"Amiraut\u00e9 de Marennes Ch\u00e2teau d' Ol\u00e9ron","count":2},{"departure":"Port d' Envaux","count":2},{"departure":"Charrou","count":2},{"departure":"Esnandres","count":2},{"departure":"Mornac","count":2},{"departure":"Saint Michel de Herm","count":2},{"departure":"Poitou Rib\u00e9rou Saujon","count":2},{"departure":"LA Rochelle","count":2},{"departure":"Riberoux de Saujon","count":2},{"departure":"R\u00e9 Ile de","count":2},{"departure":"Loix","count":2},{"departure":"Ile d' Oleron","count":2},{"departure":"la Flotte-en-R\u00e9","count":2},{"departure":"Port des Barques","count":2},{"departure":"Ars en r\u00e9","count":2},{"departure":"Riberon Riberou","count":2},{"departure":"Saint Morice Moricq","count":1},{"departure":"Mescher","count":1}]
```
### 1.2.4. data.portic.fr/api/agg/destinations?lon=lat=radius=
Les travels seulement à l'arrivée dans un rayon de radium km autour du point long/lat
### 1.2.5. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&both-to=true
Donnerait les trajets passant par le port en question, et toutes les escales.
sans doublons par défaut
avec doublons si both-to=true
### 1.2.6. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&degree=0&both-to=true
Donnerait les trajets entrants ou partants du port en question,
sans doublons, seulement les escales directement précédentes ou suivantes
sans doublons par défaut
avec doublons si both-to=true
# -*- coding: utf-8 -*-
'''
Created on 14 may 2020
@author: cplumejeaud, ggeoffroy
ANR PORTIC : used to publish data fo portic - pointcalls and travels API
This requires :
1. to have loaded data (pointcall, taxes, cargo) from navigo with LoadFilemaker.py, in schema navigo, navigocheck.
2. to have built a table ports.port_points listing all ports ( using geo_general ) with additional data and manual editing for admiralty, province.
@see navigocorpus/ETL/BuildNavigoviz.py
'''
from flask import Flask, jsonify, abort, render_template,url_for,request
from flask_cors import CORS, cross_origin
from flask_caching import Cache
import numpy as np
import csv
import json
app = Flask(__name__)
CORS(app)
app.config.from_object('config')
port = app.config['PORT']
#port = '80'
def isInside(lat, lon, radius, x, y):
"""
Will be computed by postgres at the end of the afternoon
"""
#radius is in degree in formula
if ((x - lat) * (x - lat) +
(y - lon) * (y - lon) <= radius * radius):
return True
else:
return False
def calcul_isInside(lat, lon, radius):
"""
Will be computed by postgres at the end of the afternoon
"""
import pandas as pd
df = pd.read_csv('static/data/travels_API_11mai2020.csv', sep = ';')
#radius in km is converted in radius in degree
calculated = df.apply(lambda row:isInside(lat,lon,radius/111.19492664455873,row.departure_latitude,row.departure_longitude), axis=1)
return df.loc[calculated]
@app.route('/api/pointcalls/')
def getPointcalls():
"""
Return the pointcalls as specified in API
Will be extracted from postgres, schema navigoviz, table pointcall (see navigocorpus/ETL)
"""
import pandas as pd
dfcsv = pd.read_csv('static/data/pointcalls_API_11mai2020.csv', sep = ';')
json_str = json.dumps(dfcsv.to_json(orient='records'))
return json.loads(json_str)
@app.route('/api/travels/')
def getTravels():
"""
Return the travels as specified in API
Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL),
but with a filter by default : only source_entry = from and both-from, to avoid duplicates
"""
import pandas as pd
dfcsv = pd.read_csv('static/data/travels_API_11mai2020.csv', sep = ';')
# Filter todo
json_str = json.dumps(dfcsv.to_json(orient='records'))
return json.loads(json_str)
@app.route('/api/details/departures/', methods = ['GET'])
def getDeparturesDetails():
"""
Return the travels, at the departure of the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL),
but with a filter by default : only source_entry = from and both-from, to avoid duplicates
"""
lat = float(request.args.get("lat"))
lon = float(request.args.get("lon"))
radius = int(request.args.get("radius"))
df1 = calcul_isInside(lat, lon, radius)
return json.loads(json.dumps(df1[['departure','departure_uhgs_id','departure_latitude','departure_longitude']].drop_duplicates().to_json(orient='records')))
@app.route('/api/agg/departures/', methods = ['GET'])
def getDeparturesAgg():
"""
Return the count of departures, for the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL),
but with a filter by default : only source_entry = from and both-from, to avoid duplicates
"""
lat = float(request.args.get("lat"))
lon = float(request.args.get("lon"))
radius = int(request.args.get("radius"))
df2 = calcul_isInside(lat, lon, radius).departure.value_counts().reset_index()
df2.columns = ['departure', 'count']
return json.loads(json.dumps(df2.to_json(orient='records')))
@app.route('/api/agg/destinations/', methods = ['GET'])
def getDestinationsAgg():
"""
Return the count of destinations, for the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL),
but with a filter by default : only source_entry = from and both-from, to avoid duplicates
"""
lat = float(request.args.get("lat"))
lon = float(request.args.get("lon"))
radius = int(request.args.get("radius"))
df3 = calcul_isInside(lat, lon, radius).destination_amiraute.value_counts().reset_index()
df3.columns = ['label', 'value']
df3['id'] = df3['label']
return json.loads(json.dumps(df3.to_json(orient='records')))
if __name__ == '__main__':
app.run(debug=True,port=port,threaded=True)
\ No newline at end of file
#Config variables
#port
PORT = '5004'
#cache variables
DEBUG: True
CACHE_TYPE: "simple"
CACHE_DEFAULT_TIMEOUT: 300
Click==7.0
Flask==1.1.1
Flask-Caching==1.8.0
Flask-Cors==3.0.8
geographiclib==1.50
itsdangerous==1.1.0
Jinja2==2.11.1
MarkupSafe==1.1.1
numpy==1.18.1
pandas==1.0.1
python-dateutil==2.8.1
pytz==2019.3
six==1.14.0
Werkzeug==1.0.0
This diff is collapsed.
This diff is collapsed.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment