This is a story on how to not spend even a penny by using three ETA (estimated time of arrival) services instead of one. Everything is based on my personal experience working as a back-end developer at GoDee project. GoDee is a start-up project that offers booking seats on a bus online. You could find more information about this project . here Prehistory GoDee is a public transportation service. Bus transportation by GoDee is more convenient than motorbikes common for Southeast Asia and cheaper than a taxi. The app-based system allows users to find an appropriate route, select the time, book the seat, and pay for the ride online. And one of the problems of GoDee is traffic jams that severely impact the user experience. Users get tired of waiting and get annoyed by trying to guess the bus arrival time. So, to make the commuting more convenient, it needed service to calculate the bus’s approximate arrival time, aka ETA. Developing ETA from scratch would take at least a year. So, to speed up the process, GoDee decided to implement the Google Distance Matrix API tool. Later they developed their own Pifia micro-service. Problems Over time, the business grew, and the user base increased. We encountered a problem with increasing requests in the Google Distance Matrix API. Why is this a problem? Because every request costs money, Google API provides 10.000 free queries per month, after which every 1.000 queries are charged $20. At that time, we had about 150,000 requests per month. My mentor was very dissatisfied with that. And said that system should change cashing to store ETA every 30 minutes. At that time, the system sent requests to the Google API every 3 seconds to get fresh data. However, such a cashing algorithm wasn’t efficient, since minibuses were stuck in traffic. And so the distance only changed once every ten minutes. There was another nuance. For example, five users are asking for information about the same bus, and this is the same request. The cache solved this type of problem. func newCache(cfg config.GdmCacheConfig,
	pf func( , to geometry.Coordinate) 
(durationDistancePair, error)) *Cache { := Cache{ :                  make(map[string]gdmCacheItem), :                      cfg.CacheItemTTLSec, :         cfg.InvalidationPeriodSec, : pf,
	} &res
}

func (c *Cache) get( , to geometry.Coordinate) 
(gdmCacheItem, bool) {
	c.mut.RLock()
	defer c.mut.RUnlock()

	keyStr := geometry.EncodeRawCoordinates([]geometry.Coordinate
{ , to})
	val, := c.cacheItems[keyStr] exist { val, exist
	}

	itemsWithToEq := make([]gdmCacheItem, , 
len(c.cacheItems)) _, := range c.cacheItems { v.to == to {
			itemsWithToEq = 
append(itemsWithToEq, v)
		}
	} _, := range itemsWithToEq { := 
geometry.Coordinate2Point( )
		p2 := 
geometry.Coordinate2Point(itwt.from) c.geom.DistancePointToPoint(p1, 
p2) > { } itwt, } gdmCacheItem{}, }

func (c *Cache) set( , to geometry.Coordinate) 
(gdmCacheItem, error) { := 
geometry.EncodeRawCoordinates([]geometry.Coordinate
{ , to})

	c.mut.Lock()
	defer c.mut.Unlock() v, := c.cacheItems[keyStr]; ex { v, nil
	}

	resp, := 
c.pfGetP2PDurationAndDistance( , to) err != nil { gdmCacheItem{}, err
	}

	neuItem := gdmCacheItem{ : , :   to, : durationDistancePair{ :            resp.dur, : 
resp.distanceMeters}, : 
time.Now().Add(time.Duration(c.ttlSec) * 
time.Second),
	}

	c.cacheItems[keyStr] = neuItem neuItem, nil
}

func (c *Cache) invalidate() {
	c.mut.Lock()
	defer c.mut.Unlock()
	toDelete := make([]string, , 
len(c.cacheItems)) k, := range c.cacheItems { time.Now().Before(v.invalidationTime) { }
		toDelete = append(toDelete, k)
	} _, := range toDelete { (c.cacheItems, td)
	}
}

func (c *Cache) run() { := 
time.NewTicker(time.Duration(c.invalidatePeriodSec) * time.Second) {
		select { from res cacheItems ttlSec invalidatePeriodSec pfGetP2PDurationAndDistance return from from exist if return 0 for v if for itwt p1 from if 10.0 continue return true return false from keyStr from if ex return err from if return from from to data dur distanceMeters invalidationTime return 0 for v if continue for td delete ticker for case < ()
		}
	}
} -ticker.C: c.invalidate Alternative services The cache worked, but not for long since GoDee grew even further and faced the same problem — the number of queries has increased again. It was decided to replace the Google API with OSRM. Basically, OSRM is a service for building a route based on ETA (this is a rough but the short description, if you need details, here is the ). link The Open Source Routing Machine or OSRM is a C++ implementation of a high-performance routing engine for the shortest paths in road networks. Wikipedia. OSRM has one problem: it builds routes and calculates ETA without taking traffic into account. To solve this problem, I started looking for services that can provide information about traffic in the specified part of the city. HERE Traffic was providing the data I needed. After a little study of the documentation, I wrote a small code that gets traffic information every 30 minutes. And to upload traffic information to OSRM, I wrote a small script with the command ./osrm-contract data.osrm --segment-speed-file updates.csv (more details ). here Math time: every half of the hour, there is a request to HERE to get traffic information this are two requests per hour, that is, a day is 48 requests (24 * 2 = 48) and a month is about ≈ 1.488 (48*31 = 1.488) a year 17.520. Yes, we have these free requests from HERE for 15 years would be enough. described here 
https: v_guide/topics/common-acronyms.html
type hereResponse struct {
	RWS []rws }

type rws struct {
	RW []rw }

type rw struct {
	FIS []fis }

type fis struct {
	FI []fi }

type fi struct {
	TMC tmc CF  []cf }

type tmc struct {
	PC int DE string QD string LE float64 }

type cf struct {
	TY string SP float32 SU float64 FF float64 JF float64 CN float64 }

type geocodingResponse struct {
	Response response }

type response struct {
	View []view }

type view struct {
	Result []result }

type result struct {
	MatchLevel string Location   location }

type location struct {
	DisplayPosition position }

type position struct {
	Latitude  float64 Longitude float64 }

type osmInfo struct {
	Waypoints []waypoints Code      string }

type waypoints struct {
	Nodes    []int Hint     string Distance float64 Name     string Location []float64 }

type osmDataTraffic struct {
	FromOSMID int
	ToOSMID   int
	TubeSpeed float64
	EdgeRate  float64
} containing traffic information
func CreateTrafficData(h config.TrafficConfig) 
error { := make([]osmDataTraffic, )

	x, := mercator(h.Lan, h.Lon, h.MapZoom)
	quadKey := tileXYToQuadKey(x, y, h.MapZoom)

	trafficInfo, := 
getTrafficDataToHereService(quadKey, h.APIKey) err != nil { err
	} _, := range trafficInfo.RWS[ ].RW { j := ; j < len(t.FIS[ ].FI) ; j++ {
			position, := 
getCoordinateByStreetName(t.FIS[ ].FI[j].TMC.DE, 
h.APIKey) err != nil {
				logrus.Error(err) }

			osmID, := 
requestToGetNodesOSMID(position.Latitude, 
position.Longitude, h.OSMRAddr) err != nil {
				logrus.Error(err) }

			osm = append(osm, osmDataTraffic{ : osmID[ ], :   osmID[ ], : , :  t.FIS[ ].FI[j].CF[ ].SU,
			})

		}

	} err := createCSVFile(osm); err != nil { err
	} nil
} l
func mercator(lan, lon float64, z int64) (float64, 
float64) { := lan * math.Pi / n := math.Pow( , float64(z))
	xTile := n * ((lon + ) / )
	yTile := n * ( - (math.Log(math.Tan(latRad)+ /math.Cos(latRad)) / 
math.Pi)) / xTile, yTile
} l
func tileXYToQuadKey(xTile, yTile float64, z int64) 
string { := i := uint(z); i > ; i-- { digit = mask := << (i - ) (int(xTile) & mask) != {
			digit++
		} (int(yTile) & mask) != {
			digit = digit + }
		quadKey += fmt.Sprintf( , digit)
	} quadKey
} osm id by coordinates
func requestToGetNodesOSMID(lan, lon float64, 
osrmAddr string) ([]int, error) { := osmInfo{} beginning lon And then lan url := 
fmt.Sprintf( , 
osrmAddr, lon, lan)

	resp, := http.Get(url) err != nil { nil, err
	} resp.StatusCode != http.StatusOK { nil, fmt.Errorf( , resp.StatusCode)
	}

	body, := ioutil.ReadAll(resp.Body) err != nil { nil, err
	}

	err = json.Unmarshal(body, &osm) err != nil { nil, err
	} len(osm.Waypoints) == { nil, fmt.Errorf( , lan, lon)
	} osm.Waypoints[ ].Nodes, nil
} ev_guide/topics/quick-start-geocode.html coordinates by street name

func getCoordinateByStreetName(streetName, apiKey 
string) (position, error) {

	streetName += url := 
fmt.Sprintf( , apiKey)

	gr := geocodingResponse{}

	streetNames := strings.Split(streetName, ) _, := range streetNames {
		url += s + }
	resp, := http.Get(url) err != nil { position{}, err
	} resp.StatusCode != http.StatusOK { position{}, 
fmt.Errorf( , resp.StatusCode)
	}

	body, := ioutil.ReadAll(resp.Body) err != nil { position{}, err
	}

	err = json.Unmarshal(body, &gr) err != nil { position{}, err
	} len(gr.Response.View) == { position{}, errors.New( )
	} _, := range 
gr.Response.View[ ].Result { g.MatchLevel == { g.Location.DisplayPosition, nil
		}
	} position{}, fmt.Errorf( , streetName)
}


func getTrafficDataToHereService(quadKey, apiKey 
string) (hereResponse, error) { := hereResponse{}

	url := 
fmt.Sprintf( , quadKey, 
apiKey)

	resp, := http.Get(url) err != nil { rw, err
	} resp.StatusCode != http.StatusOK { rw, fmt.Errorf( , resp.StatusCode)
	}

	body, := ioutil.ReadAll(resp.Body) err != nil { rw, err
	}

	err = json.Unmarshal(body, &rw) err != nil { rw, err
	} rw, nil
}


func createCSVFile(data []osmDataTraffic) error { err := 
os.Remove( ); err != nil {
		logrus.Error(err)
	}

	file, := 
os.Create( ) err != nil { err
	}

	defer file.Close()

	writer := csv.NewWriter(file)
	defer writer.Flush() _, := range data { := 
createArrayStringByOSMInfo(value)

		err := writer.Write(str) err != nil {
			logrus.Error(err)
		}
	} nil
}

func createArrayStringByOSMInfo(data 
osmDataTraffic) []string { str []string

	str = append(str, fmt.Sprintf( , 
data.FromOSMID))
	str = append(str, fmt.Sprintf( , 
data.ToOSMID))
	str = append(str, fmt.Sprintf( , 
data.TubeSpeed))
	str = append(str, fmt.Sprintf( , 
data.EdgeRate)) str
} // everything that these structures mean is //developer.here.com/documentation/traffic/de `json:"RWS"` `json:"RW"` `json:"FIS"` `json:"FI"` `json:"TMC"` `json:"CF"` `json:"PC"` `json:"DE"` `json:"QD"` `json:"LE"` `json:"TY"` `json:"SP"` `json:"SU"` `json:"FF"` `json:"JF"` `json:"CN"` `json:"Response"` `json:"View"` `json:"Result"` `json:"MatchLevel"` `json:"Location"` `json:"DisplayPosition"` `json:"Latitude"` `json:"Longitude"` `json:"waypoints"` `json:"code"` `json:"nodes"` `json:"hint"` `json:"distance"` `json:"name"` `json:"location"` // CreateTrafficData - function creates a cvs file osm 0 y err if return for t 0 for 0 0 -1 err 0 if continue err if continue FromOSMID 0 ToOSMID 1 TubeSpeed 0 EdgeRate 0 0 if return return // http://mathworld.wolfram.com/MercatorProjection.htm latRad 180 2 180 360 1 1 2 return // http://mathworld.wolfram.com/MercatorProjection.htm quadKey "" for 0 var 0 1 1 if 0 if 0 2 "%d" return // requestToGetNodesOSMID - function for getting osm // here it is necessary that at the // WARN only Ho Chi Minh "http://%s/nearest/v1/driving/%v,%v" err if return if return "Status code 
%d" err if return if return if 0 return "Nodes are 
empty, lan: %v, lon: %v" return 0 // https://developer.here.com/documentation/geocoder/d // getCoordinateByStreetName - function of the " Ho Chi Minh" "https://geocoder.ls.hereapi.com/6.2/ge
ocode.json?apiKey=%s&searchtext=" " " for s "+" err if return if return "Status code %d" err if return if return if 0 return "View 
response empty" for g 0 if "street" return return "street: %s 
not found" rw "https://traffic.ls.hereapi.com/traffic
/6.2/flow.json?quadkey=%s&apiKey=%s" err if return if return "Status code 
%d" err if return if return return if "./traffic/result.csv" err "./traffic/result.csv" if return for value str if return var "%v" "%v" "%v" "%v" return Preliminary tests showed that the service works perfectly, but there is a problem, HERE gives traffic information in “gibberish” and the data does not match the OSRM format. In order for the information to fit, you need to use another service HERE for geocoding + OSRM (for getting points on the map). This is approximately 450.000 requests per month. Later, OSRM was abandoned because the number of requests exceeded the free limit. We didn’t give up and enabled the HERE Distance Matrix API and temporarily removed the Google Distance Matrix API. The logic HERE is simple: we send coordinates from point A to point B and get the bus arrival time. type response struct {
	Response matrixResponse }

type matrixResponse struct {
	Route    []matrixRoute }

type matrixRoute struct {
	Summary summary }

type summary struct {
	Distance    int TrafficTime int }

func HereDistanceETA() (response, error) { := response{}
	query := fmt.Sprintf( , , .Lat, .Lon)
	query += fmt.Sprintf( , , to.Lat, 
to.Lon)
	query += url := fmt.Sprintf( , h.hereAPIKey)
	url += query
	resp, := http.Get(url) err != nil {
		logrus.WithFields(logrus.Fields{ :   url, : err,
		}).Error( ) durationDistancePair{}, err
	} resp.StatusCode != http.StatusOK { durationDistancePair{}, 
fmt.Errorf( , 
resp.StatusCode)
	}

	body, := ioutil.ReadAll(resp.Body) err != nil { durationDistancePair{}, err
	}

	err = json.Unmarshal(body, &matrixResponse) err != nil { durationDistancePair{}, err
	} len(matrixResponse.Response.Route) == { durationDistancePair{}, 
errors.New( )
	}
	res := durationDistancePair{ :            
time.Duration(matrixResponse.Response.Route[ ].Summ
ary.TrafficTime) * time.Second, : 
matrixResponse.Response.Route[ ].Summary.Distance,
	} res, nil
} `json:"response"` `json:"route"` `json:"summary"` `json:"distance"` `json:"trafficTime"` matrixResponse "&waypoint%v=geo!%v,%v" 0 from from "&waypoint%v=geo!%v,%v" 1 "&mode=fastest;car;traffic:enabled" "https://route.ls.hereapi.com/routing/7
.2/calculateroute.json?apiKey=%s" err if "url" "error" "Get here response 
failed" return if return "Here service, status code %d" err if return if return if 0 return "Matrix response empty" dur 0 distanceMeters 0 return After we installed everything on the test server and started checking, we received the first feedback from the testers. They said that ETA reads the time incorrectly. We started looking for the problem, looked at logs (we used Data dog for logs), logs, and tests showed that everything works perfectly. We decided to ask about the problem in a little more detail, and it turned out that if the car is in traffic for 15 minutes, ETA shows the same time. We decided that this is because of the cache because it stores the original time and does not update it for 30 minutes. We started looking for the problem, at the beginning we checked the data on the web version of the HERE Distance Matrix API (which is called we go here), everything worked fine, we received the same ETA. This problem was also checked on the google map service. There was no problem. The services themselves show this ETA. We explained everything to testers and businesses, and they accepted everything. Our team lead suggested connecting another ETA service and returning the Google API as a backup option and writing code with the logic of switching services (the switch was needed if the requests pass the free number of requests). The code works the following way: val = getCount() used getMax() <= val { free requests the service used
newService = switchService(s) reached, the service newService( , to) the service // getting the number of queries if // checking for the limit of for // // if the limit is switch return return from // giving the logic of new We found the following Mapbox service, connected it, installed it, and it worked. As a result, our ETA had: “Here” — 250,000 free requests per month Google — 10,000 free requests per month Mapbox — 100,000 free requests per month Conclusion Always look for alternatives, sometimes it happens that the business doesn't want to pay the money for the service and refuses it. As a developer who has worked hard on the service, you should bring the task to real use. This article describes how we were trying to connect more services for the free use of ETA because the business did not want to pay for the service. P.S. As a developer, I believe that if the tool is good and does its job well, then you can pay for the tool’s services (or find Open source projects :D). Previously published at https://blog.maddevs.io/how-to-make-three-paid-eta-services-one-free-6edc6affface

Processing 350K Requests Per Month via Three Free ETA Services Instead of One Paid Google Service

Too Long; Didn't Read

Mad Devs

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...