First code of api, without the postgres backend for the moment. It will come...

First code of api, without the postgres backend for the moment. It will come soon. Many ideas to test to enhance performance.

porticapi/README.md

+
+- [1. Portic data API : get dynamic data from database](#1-portic-data-api--get-dynamic-data-from-database)
+  - [1.1. Installation](#11-installation)
+    - [1.1.1. Running without Apache](#111-running-without-apache)
+    - [1.1.2. Running with Apache](#112-running-with-apache)
+    - [1.1.3. Debug things](#113-debug-things)
+  - [1.2. List of possible requests](#12-list-of-possible-requests)
+    - [1.2.1. data.portic.fr/api/travels?](#121-dataporticfrapitravels)
+    - [1.2.2. data.portic.fr/api/pointcalls?](#122-dataporticfrapipointcalls)
+    - [1.2.3. data.portic.fr/api/details/departures?lon=lat=radius=](#123-dataporticfrapidetailsdepartureslonlatradius)
+    - [1.2.4. data.portic.fr/api/agg/destinations?lon=lat=radius=](#124-dataporticfrapiaggdestinationslonlatradius)
+    - [1.2.5. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&both-to=true](#125-dataporticfrapitravelslinktoport%22uhgsid-du-port%22both-totrue)
+    - [1.2.6. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&degree=0&both-to=true](#126-dataporticfrapitravelslinktoport%22uhgsid-du-port%22degree0both-totrue)
+
+
+
+# 1. Portic data API : get dynamic data from database
+
+Web API for data above the postgres database developped with Flask
+
+![](temps_telechargement.png)
+
+## 1.1. Installation
+
+Neep Python 3 to be installed
+
+Clone or download this git repo, then
+
+pip3 install -r requirements.txt
+
+### 1.1.1. Running without Apache
+
+python apidata.py
+
+(or create a virtualenv, then activate it and pip install -r requirements && python apidata.py)
+
+Backend is running and serve the data on http://localhost:5004 (change the PORT variable in config.py)
+
+### 1.1.2. Running with Apache
+
+todo 
+
+### 1.1.3. Debug things
+
+C:\Python37\lib\site-packages\flask\app.py:2446: DtypeWarning: Columns (31) have mixed types.Specify dtype option on import or set low_memory=False.
+
+
+## 1.2. List of possible requests
+
+
+### 1.2.1. data.portic.fr/api/travels?
+-- TOUT en json
+
+### 1.2.2. data.portic.fr/api/pointcalls?
+-- TOUT en json
+
+### 1.2.3. data.portic.fr/api/details/departures?lon=lat=radius=
+
+Les travels seulement au départ dans un rayon de radium km autour du point long/lat
+
+Exemple
+`http://127.0.0.1:5004/api/agg/departures/?lat=46&lon=-1&radius=100`
+
+```JS
+[{"departure":"La Rochelle","count":2631},{"departure":"Marennes","count":1996},{"departure":"Rochefort","count":894},{"departure":"Aligre de Marans","count":705},{"departure":"Saint Martin de R\u00e9","count":673},{"departure":"Ol\u00e9ron","count":600},{"departure":"Rib\u00e9rou Saujon","count":595},{"departure":"Ars en R\u00e9","count":460},{"departure":"La Tremblade","count":320},{"departure":"La Flotte en R\u00e9","count":289},{"departure":"Moricq","count":230},{"departure":"Aligre","count":155},{"departure":"Le Chateau d' Ol\u00e9ron","count":131},{"departure":"La Perrotine","count":121},{"departure":"Oleron","count":115},{"departure":"Saint Denis d' Ol\u00e9ron","count":102},{"departure":"Saint Savinien","count":100},{"departure":"Chaillevette","count":92},{"departure":"Marans","count":82},{"departure":"Olleron","count":79},{"departure":"Saint Michel en l' Herm","count":77},{"departure":"Ile de R\u00e9","count":76},{"departure":"Soubise","count":66},{"departure":"Esnandes","count":62},{"departure":"La Tranche sur Mer","count":56},{"departure":"Seudre","count":54},{"departure":"\u00eele de R\u00e9","count":38},{"departure":"Royan","count":31},{"departure":"Talmont Talmont Saint Hilaire","count":31},{"departure":"\u00cele d' Ol\u00e9ron","count":26},{"departure":"Ile d' Ol\u00e9ron","count":26},{"departure":"Port d' Envaux","count":22},{"departure":"Brouage","count":22},{"departure":"\u00cele de R\u00e9","count":20},{"departure":"Mortagne","count":17},{"departure":"Le Plomb","count":16},{"departure":"Saint Martin de Rh\u00e9","count":12},{"departure":"Loix en R\u00e9","count":12},{"departure":"L' Houm\u00e9e","count":12},{"departure":"la Rochelle","count":11},{"departure":"Taillebourg","count":10},{"departure":"dans le Pertuis","count":10},{"departure":"Dercie","count":8},{"departure":"Seudres","count":8},{"departure":"Saintes","count":8},{"departure":"Martroux","count":8},{"departure":"Bourcefranc","count":6},{"departure":"La Tranche sur mer","count":6},{"departure":"Lu\u00e7on","count":6},{"departure":"La Pr\u00e9e","count":6},{"departure":"Champagn\u00e9 sur Marais","count":5},{"departure":"Fouras","count":4},{"departure":"d' Aligre","count":4},{"departure":"Saint Martin","count":4},{"departure":"La tremblade","count":4},{"departure":"La Perotine","count":4},{"departure":"La Flotte","count":4},{"departure":"Martrou","count":4},{"departure":"la Tremblade","count":4},{"departure":"Marenne","count":4},{"departure":"Courreaux d' Ol\u00e9ron","count":4},{"departure":"Ars En R\u00e9","count":3},{"departure":"la Perrotine","count":3},{"departure":"Ars","count":3},{"departure":"Auvaux Port d' Envaux","count":2},{"departure":"\u00cele d' Oleron","count":2},{"departure":"Saint Michel en Poitou Saint Michel en l' Herm","count":2},{"departure":"Courraux d' Ol\u00e9ron","count":2},{"departure":"R\u00e9","count":2},{"departure":"La Seudre","count":2},{"departure":"Saint Trojan en Ol\u00e9ron","count":2},{"departure":"Rib\u00e9rou","count":2},{"departure":"La B\u00e9chel La B\u00e9chade","count":2},{"departure":"La Tranche","count":2},{"departure":"Aiguillon","count":2},{"departure":"dans Le Pertuis","count":2},{"departure":"Oll\u00e9ron","count":2},{"departure":"Rade de La Pallice","count":2},{"departure":"Coureaux de R\u00e9","count":2},{"departure":"Daligre","count":2},{"departure":"Saint Martin en R\u00e9","count":2},{"departure":"Ile de Re","count":2},{"departure":"L' Aiguillon","count":2},{"departure":"Saint Denis \u00cele d' Ol\u00e9ron","count":2},{"departure":"Sudre","count":2},{"departure":"Le Pr\u00e9e","count":2},{"departure":"Amiraut\u00e9 de Marennes Ch\u00e2teau d' Ol\u00e9ron","count":2},{"departure":"Port d' Envaux","count":2},{"departure":"Charrou","count":2},{"departure":"Esnandres","count":2},{"departure":"Mornac","count":2},{"departure":"Saint Michel de Herm","count":2},{"departure":"Poitou Rib\u00e9rou Saujon","count":2},{"departure":"LA Rochelle","count":2},{"departure":"Riberoux de Saujon","count":2},{"departure":"R\u00e9 Ile de","count":2},{"departure":"Loix","count":2},{"departure":"Ile d' Oleron","count":2},{"departure":"la Flotte-en-R\u00e9","count":2},{"departure":"Port des Barques","count":2},{"departure":"Ars en r\u00e9","count":2},{"departure":"Riberon Riberou","count":2},{"departure":"Saint Morice Moricq","count":1},{"departure":"Mescher","count":1}]
+```
+
+### 1.2.4. data.portic.fr/api/agg/destinations?lon=lat=radius=
+
+Les travels seulement à l'arrivée dans un rayon de radium km autour du point long/lat
+
+
+### 1.2.5. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&both-to=true
+
+Donnerait les trajets passant par le port en question, et toutes les escales. 
+sans doublons par défaut
+avec doublons si both-to=true
+
+### 1.2.6. data.portic.fr/api/travels?link_to_port="UHGS_id du port"&degree=0&both-to=true
+
+Donnerait les trajets entrants ou partants du port en question, 
+sans doublons, seulement les escales directement précédentes ou suivantes 
+sans doublons par défaut
+avec doublons si both-to=true
+
+
+

porticapi/apidata.py

+# -*- coding: utf-8 -*-
+'''
+Created on 14 may 2020
+@author: cplumejeaud, ggeoffroy
+ANR PORTIC : used to publish data fo portic - pointcalls and travels API
+This requires :
+1. to have loaded data (pointcall, taxes, cargo) from navigo with LoadFilemaker.py, in schema navigo, navigocheck.
+2. to have built a table ports.port_points listing all ports ( using geo_general ) with additional data and manual editing for admiralty, province.
+@see navigocorpus/ETL/BuildNavigoviz.py
+'''
+
+from flask import Flask, jsonify, abort, render_template,url_for,request
+from flask_cors import CORS, cross_origin
+from flask_caching import Cache
+import numpy as np
+import csv
+import json
+
+app = Flask(__name__)
+CORS(app)
+
+app.config.from_object('config')
+port = app.config['PORT']      
+#port = '80'
+
+def isInside(lat, lon, radius, x, y):    
+    """
+    Will be computed by postgres at the end of the afternoon
+    """
+    #radius is in degree in formula   
+    if ((x - lat) * (x - lat) + 
+        (y - lon) * (y - lon) <= radius * radius): 
+        return True
+    else: 
+        return False
+
+def calcul_isInside(lat, lon, radius):
+    """
+    Will be computed by postgres at the end of the afternoon
+    """
+    import pandas as pd
+    df = pd.read_csv('static/data/travels_API_11mai2020.csv', sep = ';')
+    #radius in km is converted in radius in degree
+    calculated = df.apply(lambda row:isInside(lat,lon,radius/111.19492664455873,row.departure_latitude,row.departure_longitude), axis=1)
+    return df.loc[calculated]
+
+@app.route('/api/pointcalls/')   
+def getPointcalls():
+    """
+    Return the pointcalls as specified in API
+    Will be extracted from postgres, schema navigoviz, table pointcall (see navigocorpus/ETL)
+    """
+    import pandas as pd
+    dfcsv = pd.read_csv('static/data/pointcalls_API_11mai2020.csv', sep = ';')
+    json_str = json.dumps(dfcsv.to_json(orient='records'))
+    return json.loads(json_str)
+
+@app.route('/api/travels/')   
+def getTravels():
+    """
+    Return the travels as specified in API
+    Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL), 
+    but with a filter by default : only source_entry = from and both-from, to avoid duplicates
+    """
+    import pandas as pd
+    dfcsv = pd.read_csv('static/data/travels_API_11mai2020.csv', sep = ';')
+    # Filter todo
+    json_str = json.dumps(dfcsv.to_json(orient='records'))
+    return json.loads(json_str)
+
+@app.route('/api/details/departures/', methods = ['GET'])   
+def getDeparturesDetails():
+    """
+    Return the travels, at the departure of the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
+    Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL), 
+    but with a filter by default : only source_entry = from and both-from, to avoid duplicates
+    """
+    lat = float(request.args.get("lat"))
+    lon = float(request.args.get("lon"))
+    radius = int(request.args.get("radius"))
+    df1 = calcul_isInside(lat, lon, radius)
+    return json.loads(json.dumps(df1[['departure','departure_uhgs_id','departure_latitude','departure_longitude']].drop_duplicates().to_json(orient='records')))
+
+@app.route('/api/agg/departures/', methods = ['GET']) 
+def getDeparturesAgg():
+    """
+    Return the count of departures, for the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
+    Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL), 
+    but with a filter by default : only source_entry = from and both-from, to avoid duplicates
+    """
+    lat = float(request.args.get("lat"))
+    lon = float(request.args.get("lon"))
+    radius = int(request.args.get("radius"))
+    df2 = calcul_isInside(lat, lon, radius).departure.value_counts().reset_index()
+    df2.columns = ['departure', 'count']
+    return json.loads(json.dumps(df2.to_json(orient='records')))   
+
+@app.route('/api/agg/destinations/', methods = ['GET'])   
+def getDestinationsAgg():
+    """
+    Return the count of destinations, for the points located in a 100 km radius neighbourhood from the lat/lon given in parameter
+    Will be extracted from postgres, schema navigoviz, table built_travels (see navigocorpus/ETL), 
+    but with a filter by default : only source_entry = from and both-from, to avoid duplicates
+    """
+    lat = float(request.args.get("lat"))
+    lon = float(request.args.get("lon"))
+    radius = int(request.args.get("radius"))
+    df3 = calcul_isInside(lat, lon, radius).destination_amiraute.value_counts().reset_index()
+    df3.columns = ['label', 'value']
+    df3['id'] = df3['label']
+    return json.loads(json.dumps(df3.to_json(orient='records')))            
+
+
+
+if __name__ == '__main__':
+    app.run(debug=True,port=port,threaded=True)  
\ No newline at end of file

porticapi/config.py

+#Config variables
+#port
+PORT = '5004'
+#cache variables
+DEBUG: True        
+CACHE_TYPE: "simple"
+CACHE_DEFAULT_TIMEOUT: 300

porticapi/requirements.txt

+Click==7.0
+Flask==1.1.1
+Flask-Caching==1.8.0
+Flask-Cors==3.0.8
+geographiclib==1.50
+itsdangerous==1.1.0
+Jinja2==2.11.1
+MarkupSafe==1.1.1
+numpy==1.18.1
+pandas==1.0.1
+python-dateutil==2.8.1
+pytz==2019.3
+six==1.14.0
+Werkzeug==1.0.0